Skip to main content
SpringerLink
Log in

Deep learning-based detection of motion artifacts in probe-based confocal laser endomicroscopy images

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose:

Probe-based confocal laser endomicroscopy (pCLE) is a subcellular in vivo imaging technique capable of producing images that enable diagnosis of malign structural modifications in epithelial tissue. Images acquired with pCLE are, however, often tainted by significant artifacts that impair diagnosis. This is especially detrimental for automated image analysis, which is why said images are often excluded from recognition pipelines.

Methods

We present an approach for the automatic detection of motion artifacts in pCLE images and apply this methodology to a data set of 15 thousand images of epithelial tissue acquired in the oral cavity and the vocal folds. The approach is based on transfer learning from intermediate endpoints within a pre-trained Inception v3 network with tailored preprocessing. For detection within the non-rectangular pCLE images, we perform pooling within the activation maps of the network and evaluate this at different network depths.

Results

We achieved area under the ROC curve values of 0.92 with the proposed method, compared to 0.80 for the best feature-based machine learning approach. Our overall accuracy with the presented approach is 94.8%.

Conclusion

Over traditional machine learning approaches with state-of-the-art features, we achieved significantly improved overall performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://www5.cs.fau.de/~aubreville/.

References

  1. Abbaci M, Breuskin I, Casiraghi O, De Leeuw F (2014) Confocal laser endomicroscopy for non-invasive head and neck cancer imaging: a comprehensive review. Oral Oncol 50(8):711–6. https://doi.org/10.1016/j.oraloncology.2014.05.002

    Article  PubMed  Google Scholar 

  2. Agaimy A, Weichert W (2016) Grading of head and neck neoplasms. Der Pathol 37(4):285–292. https://doi.org/10.1007/s00292-016-0173-9

    Article  CAS  Google Scholar 

  3. Araujo H, Dias JM (1996) An introduction to the log-polar mapping (image sampling). In: Proceedings 2nd workshop on cybernetic vision. IEEE, pp 139–144. https://doi.org/10.1109/CYBVIS.1996.629454

  4. Aubreville M, Knipfer C, Oetter N, Jaremenko Christian, Rodner E, Denzler J, Bohr C, Neumann H, Stelzle F, Maier A (2017) Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning. Sci Rep 7(1):11979. https://doi.org/10.1038/s41598-017-12320-8

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Bier B, Mualla F, Steidl S, Bohr C, Neumann H, Maier A, Hornegger J (2015) Band-pass filter design by segmentation in frequency domain for detection of epithelial cells in endomicroscope images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 413–418. https://doi.org/10.1007/978-3-662-46224-9_71

  6. Chauhan SS, Dayyeh BKA, Bhat YM, Gottlieb KT, Hwang JH, Komanduri S, Konda V, Lo SK, Manfredi MA, Maple JT, Murad FM, Siddiqui UD, Banerjee S, Wallace MB (2014) Confocal laser endomicroscopy. Gastrointest Endosc 80(6):928–938. https://doi.org/10.1016/j.gie.2014.06.021

    Article  Google Scholar 

  7. Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013 16(Pt 2):411–418. https://doi.org/10.1007/978-3-642-40763-5_51

  8. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, CVPR 2005. 1:886–893. https://doi.org/10.1109/CVPR.2005.177

  9. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 248–255. https://doi.org/10.1109/CVPR.2009.5206848

  10. Deniz O, García GB, Salido J, De la Torre F (2011) Face recognition using histograms of oriented gradients. Pattern Recognit Lett 32(12):1598–1603. https://doi.org/10.1016/j.patrec.2011.01.004

    Article  Google Scholar 

  11. Dittberner A, Rodner E, Ortmann W, Stadler J, Schmidt C, Petersen I, Stallmach A, Denzler J, Guntinas-Lichius O (2016) Automated analysis of confocal laser endomicroscopy images to detect head and neck cancer. Head Neck 38(S1):E1419–E1426. https://doi.org/10.1002/hed.24253

    Article  PubMed  Google Scholar 

  12. Gil D, Ramos-Terrades O, Minchole E, Sanchez C, de Frutos NC, Diez-Ferrer M, Ortiz RM, Rosell A (2017) Classification of confocal endomicroscopy patterns for diagnosis of lung cancer. In: Medical image computing and computer-assisted intervention—MICCAI 2017, pp. 151–159. Springer, Cham. https://doi.org/10.1007/978-3-319-67543-5_15

  13. Goncalves M, Iro H, Dittberner A, Agaimy A, Bohr C (2017) Value of confocal laser endomicroscopy in the diagnosis of vocal cord lesions. Eur Rev Med Pharmacol Sci 21:3990–3997

    CAS  PubMed  Google Scholar 

  14. Hong J, Park By, Park H (2017) Convolutional neural network classifier for distinguishing Barrett’s esophagus and neoplasia endomicroscopy images. In: 39th Annual International conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 2892–2895. https://doi.org/10.1109/EMBC.2017.8037461

  15. Izadyyazdanabadi M, Belykh E, Cavallo C, Zhao X, Gandhi S, Moreira LB, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Weakly-supervised learning-based feature localization in confocal laser endomicroscopy glioma images. arXiv preprint arXiv:1804.09428v1

  16. Izadyyazdanabadi M, Belykh E, Martirosyan N, Eschbacher J, Nakaji P, Yang Y, Preul MC (2017)Improving utility of brain tumor confocal laser endomicroscopy - objective value assessment and diagnostic frame detection with convolutional neural networks. In: Medical Imaging 2017, vol 10134. SPIE. https://doi.org/10.1117/12.2254902

  17. Izadyyazdanabadi M, Belykh E, Mooney M, Eschbacher J, Nakaji P, Yang Y, Preul MC (2018) Prospects for theranostics in neurosurgical technology—empowering confocal laser endomicroscopy diagnostics via deep learning. arXiv preprint arXiv:1804.09873

  18. Izadyyazdanabadi M, Belykh E, Mooney M, Martirosyan N, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Convolutional neural networks: ensemble modeling, fine-tuning and unsupervised semantic localization for neurosurgical cle images. J Vis Commun Image Represent 54:10–20. https://doi.org/10.1016/j.jvcir.2018.04.004

    Article  Google Scholar 

  19. Jaremenko C, Maier A, Steidl S, Hornegger J, Oetter N, Knipfer C, Stelzle F, Neumann H (2015) Classification of confocal laser endomicroscopic images of the oral cavity to distinguish pathological from healthy tissue. In: Bildverarbeitung für die Medizin 2015. Springer, Berlin, pp 479–485. https://doi.org/10.1007/978-3-662-46224-9_82

  20. Kingma D, Ba J (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

  21. Laemmel E, Genet M, Le Goualher G, Perchant A, Le Gargasson JF, Vicaut E (2004) Fibered confocal fluorescence microscopy (Cell-viZio) facilitates extended imaging in the field of microcirculation. A comparison with intravital microscopy. J Vasc Res 41(5):400–411. https://doi.org/10.1159/000081209

    Article  PubMed  Google Scholar 

  22. Macé V, Ahluwalia A, Coron E, Le Rhun M, Boureille A, Bossard C, Mosnier JF, Matysiak-Budnik T, Tarnawski AS (2015) Confocal laser endomicroscopy: a new gold standard for the assessment of mucosal healing in ulcerative colitis. J Gastroenterol Hepatol 30:85–92. https://doi.org/10.1111/jgh.12748

    Article  CAS  PubMed  Google Scholar 

  23. Maier AK, Schebesch F, Syben C, Würfl T, Steidl S, Choi JH, Fahrig R (2017) Precision learning: towards use of known operators in neural networks. arXiv preprint arXiv:1712.00374

  24. Maier H, Dietz A, Gewelke U, Heller W, Weidauer H (1992) Tobacco and alcohol and the risk of head and neck cancer. Clin Investig 70(3–4):320–327. https://doi.org/10.1007/BF00184668

    Article  CAS  PubMed  Google Scholar 

  25. Muto M, Nakane M, Katada C, Sano Y, Ohtsu A, Esumi H, Ebihara S, Yoshida S (2004) Squamous cell carcinoma in situ at oropharyngeal and hypopharyngeal mucosal sites. Cancer 101(6):1375–1381. https://doi.org/10.1002/cncr.20482

    Article  PubMed  Google Scholar 

  26. Nachalon Y, Alkan U, Shvero J, Yaniv D, Shkedy Y, Limon D, Popovtzer A (2017) Assessment of laryngeal cancer in patients younger than 40 years. The Laryngoscope. https://doi.org/10.1002/lary.26951

    Article  PubMed  Google Scholar 

  27. Neumann H, Langner C, Neurath MF, Vieth M (2012) Confocal laser endomicroscopy for diagnosis of Barrett’s Esophagus. Front Oncol 2:42. https://doi.org/10.3389/fonc.2012.00042

    Article  PubMed  PubMed Central  Google Scholar 

  28. Neumann H, Vieth M, Atreya R, Neurath MF, Mudter J (2011) Prospective evaluation of the learning curve of confocal laser endomicroscopy in patients with IBD. Histol Histopathol 26(7):867–872

    PubMed  Google Scholar 

  29. Oetter N, Knipfer C, Rohde M, Wilmowsky C, Maier A, Brunner K, Adler W, Neukam FW, Neumann H, Stelzle F (2016) Development and validation of a classification and scoring system for the diagnosis of oral squamous cell carcinomas through confocal laser endomicroscopy. J Transl Med 14(1):1–11. https://doi.org/10.1186/s12967-016-0919-4

    Article  Google Scholar 

  30. Oquab M, Bottou L, Laptev I, Sivic J (2015) Is object localization for free? Weakly-supervised learning with convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, pp 685–694. https://doi.org/10.1109/CVPR.2015.7298668

  31. Parikh ND, Gibson J, Nagar A, Ahmed AA, Aslanian HR (2016) Confocal laser endomicroscopy features of sessile serrated adenomas/polyps. U Eur Gastroenterol J 4(4):599–603. https://doi.org/10.1177/2050640615621819

    Article  CAS  Google Scholar 

  32. Pavlov V, Meyronet D, Meyer-Bisch V, Armoiry X, Pikul B, Dumot C, Beuriat P-A, Signorelli F, Guyotat J (2016) Intraoperative probe-based confocal laser endomicroscopy in surgery and stereotactic biopsy of low-grade and high-grade gliomas. Neurosurgery 79(4):604–612. https://doi.org/10.1227/NEU.0000000000001365

    Article  PubMed  Google Scholar 

  33. Robert Koch Institut. Zentrum für Krebsregisterdaten (2017) Krebs in Deutschland für 2013/2014, 11th edn. Robert Koch Institut, Berlin

  34. Stoeve M, Aubreville M, Oetter N, Knipfer C, Neumann H, Stelzle F, Maier A (2018) Motion Artifact Detection in Confocal Laser Endomicroscopy Images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 328–333. https://doi.org/10.1007/978-3-662-56537-7_85

  35. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR.2015.7298594

  36. Vo K, Jaremenko Christian, Bohr C, Neumann H, Maier A (2017) Automatic classification and pathological staging of confocal laser endomicroscopic images of the vocal cords. In: Bildverarbeitung für die Medizin 2017. Springer, Berlin, pp 312–317. https://doi.org/10.1007/978-3-662-54345-0_70

  37. Westra WH (2015) The pathology of HPV-related head and neck cancer: implications for the diagnostic pathologist. Semin Diagn Pathol 32(1):42–53. https://doi.org/10.1053/j.semdp.2015.02.023

    Article  PubMed  Google Scholar 

  38. Wiesner C, Jäger W, Salzer A, Biesterfeld S, Kiesslich R, Hampel C, Thüroff JW, Goetz M (2010) Confocal laser endomicroscopy for the diagnosis of urothelial bladder neoplasia: a technology of the future? BJU Int 107(3):399–403. https://doi.org/10.1111/j.1464-410X.2010.09540.x

    Article  PubMed  Google Scholar 

  39. Xing F, Xie Y, Su H, Liu F, Yang L (2017) Deep learning in microscopy image analysis: a survey. IEEE Trans Neural Netw Learn Syst PP(99):1–19. https://doi.org/10.1109/TNNLS.2017.2766168

  40. Yang Y, Li T, Li W, Wu H, Fan W, Zhang W (2017) Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks. In: Medical image computing and computer-assisted intervention—MICCAI 2017. Springer, Cham, pp 533–540. https://doi.org/10.1007/978-3-319-66179-7_61

Download references

Author information

Authors and Affiliations

  1. Pattern Recognition Lab, Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Marc Aubreville, Maike Stoeve & Andreas Maier

  2. Department of Oral and Maxillofacial Surgery, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Nicolai Oetter & Florian Stelzle

  3. Department of Otorhinolaryngology, Head and Neck Surgery, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Miguel Goncalves

  4. Department of Oral and Maxillofacial Surgery, University Medical Center Hamburg-Eppendorf, Universität Hamburg, Hamburg, Germany

    Christian Knipfer

  5. Department of Otorhinolaryngology, Head and Neck Surgery, University Hospital, Universität Regensburg, Regensburg, Germany

    Christopher Bohr

  6. First Department of Internal Medicine, University Hospital Mainz, Johannes Gutenberg-Universität Mainz, Mainz, Germany

    Helmut Neumann

Authors
  1. Marc Aubreville
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maike Stoeve
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolai Oetter
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Miguel Goncalves
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christian Knipfer
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Helmut Neumann
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Christopher Bohr
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Florian Stelzle
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Andreas Maier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marc Aubreville.

Ethics declarations

Conflict of interest

All authors declare that they do not have conflicts of interest regarding the work covered by this manuscript.

Human and animal participants

All procedures involving human participants were in accordance with the 1964 Declaration of Helsinki and its later amendments.

Informed consent

The local ethics committee approved the studies (ethics committee of the University of Erlangen-Nürnberg; reference numbers 243 12 B and 60 14 B), and all patients gave their written informed consent.

Additional information

Marc Aubreville and Maike Stoeve have been contributing equally to the research in this paper.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aubreville, M., Stoeve, M., Oetter, N. et al. Deep learning-based detection of motion artifacts in probe-based confocal laser endomicroscopy images. Int J CARS 14, 31–42 (2019). https://doi.org/10.1007/s11548-018-1836-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-018-1836-1

Keywords

Search

Navigation