Abstract
Purpose:
Probe-based confocal laser endomicroscopy (pCLE) is a subcellular in vivo imaging technique capable of producing images that enable diagnosis of malign structural modifications in epithelial tissue. Images acquired with pCLE are, however, often tainted by significant artifacts that impair diagnosis. This is especially detrimental for automated image analysis, which is why said images are often excluded from recognition pipelines.
Methods
We present an approach for the automatic detection of motion artifacts in pCLE images and apply this methodology to a data set of 15 thousand images of epithelial tissue acquired in the oral cavity and the vocal folds. The approach is based on transfer learning from intermediate endpoints within a pre-trained Inception v3 network with tailored preprocessing. For detection within the non-rectangular pCLE images, we perform pooling within the activation maps of the network and evaluate this at different network depths.
Results
We achieved area under the ROC curve values of 0.92 with the proposed method, compared to 0.80 for the best feature-based machine learning approach. Our overall accuracy with the presented approach is 94.8%.
Conclusion
Over traditional machine learning approaches with state-of-the-art features, we achieved significantly improved overall performance.
Similar content being viewed by others
References
Abbaci M, Breuskin I, Casiraghi O, De Leeuw F (2014) Confocal laser endomicroscopy for non-invasive head and neck cancer imaging: a comprehensive review. Oral Oncol 50(8):711–6. https://doi.org/10.1016/j.oraloncology.2014.05.002
Agaimy A, Weichert W (2016) Grading of head and neck neoplasms. Der Pathol 37(4):285–292. https://doi.org/10.1007/s00292-016-0173-9
Araujo H, Dias JM (1996) An introduction to the log-polar mapping (image sampling). In: Proceedings 2nd workshop on cybernetic vision. IEEE, pp 139–144. https://doi.org/10.1109/CYBVIS.1996.629454
Aubreville M, Knipfer C, Oetter N, Jaremenko Christian, Rodner E, Denzler J, Bohr C, Neumann H, Stelzle F, Maier A (2017) Automatic classification of cancerous tissue in laserendomicroscopy images of the oral cavity using deep learning. Sci Rep 7(1):11979. https://doi.org/10.1038/s41598-017-12320-8
Bier B, Mualla F, Steidl S, Bohr C, Neumann H, Maier A, Hornegger J (2015) Band-pass filter design by segmentation in frequency domain for detection of epithelial cells in endomicroscope images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 413–418. https://doi.org/10.1007/978-3-662-46224-9_71
Chauhan SS, Dayyeh BKA, Bhat YM, Gottlieb KT, Hwang JH, Komanduri S, Konda V, Lo SK, Manfredi MA, Maple JT, Murad FM, Siddiqui UD, Banerjee S, Wallace MB (2014) Confocal laser endomicroscopy. Gastrointest Endosc 80(6):928–938. https://doi.org/10.1016/j.gie.2014.06.021
Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013 16(Pt 2):411–418. https://doi.org/10.1007/978-3-642-40763-5_51
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, CVPR 2005. 1:886–893. https://doi.org/10.1109/CVPR.2005.177
Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 248–255. https://doi.org/10.1109/CVPR.2009.5206848
Deniz O, García GB, Salido J, De la Torre F (2011) Face recognition using histograms of oriented gradients. Pattern Recognit Lett 32(12):1598–1603. https://doi.org/10.1016/j.patrec.2011.01.004
Dittberner A, Rodner E, Ortmann W, Stadler J, Schmidt C, Petersen I, Stallmach A, Denzler J, Guntinas-Lichius O (2016) Automated analysis of confocal laser endomicroscopy images to detect head and neck cancer. Head Neck 38(S1):E1419–E1426. https://doi.org/10.1002/hed.24253
Gil D, Ramos-Terrades O, Minchole E, Sanchez C, de Frutos NC, Diez-Ferrer M, Ortiz RM, Rosell A (2017) Classification of confocal endomicroscopy patterns for diagnosis of lung cancer. In: Medical image computing and computer-assisted intervention—MICCAI 2017, pp. 151–159. Springer, Cham. https://doi.org/10.1007/978-3-319-67543-5_15
Goncalves M, Iro H, Dittberner A, Agaimy A, Bohr C (2017) Value of confocal laser endomicroscopy in the diagnosis of vocal cord lesions. Eur Rev Med Pharmacol Sci 21:3990–3997
Hong J, Park By, Park H (2017) Convolutional neural network classifier for distinguishing Barrett’s esophagus and neoplasia endomicroscopy images. In: 39th Annual International conference of the IEEE engineering in medicine and biology society (EMBC). IEEE, pp 2892–2895. https://doi.org/10.1109/EMBC.2017.8037461
Izadyyazdanabadi M, Belykh E, Cavallo C, Zhao X, Gandhi S, Moreira LB, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Weakly-supervised learning-based feature localization in confocal laser endomicroscopy glioma images. arXiv preprint arXiv:1804.09428v1
Izadyyazdanabadi M, Belykh E, Martirosyan N, Eschbacher J, Nakaji P, Yang Y, Preul MC (2017)Improving utility of brain tumor confocal laser endomicroscopy - objective value assessment and diagnostic frame detection with convolutional neural networks. In: Medical Imaging 2017, vol 10134. SPIE. https://doi.org/10.1117/12.2254902
Izadyyazdanabadi M, Belykh E, Mooney M, Eschbacher J, Nakaji P, Yang Y, Preul MC (2018) Prospects for theranostics in neurosurgical technology—empowering confocal laser endomicroscopy diagnostics via deep learning. arXiv preprint arXiv:1804.09873
Izadyyazdanabadi M, Belykh E, Mooney M, Martirosyan N, Eschbacher J, Nakaji P, Preul MC, Yang Y (2018) Convolutional neural networks: ensemble modeling, fine-tuning and unsupervised semantic localization for neurosurgical cle images. J Vis Commun Image Represent 54:10–20. https://doi.org/10.1016/j.jvcir.2018.04.004
Jaremenko C, Maier A, Steidl S, Hornegger J, Oetter N, Knipfer C, Stelzle F, Neumann H (2015) Classification of confocal laser endomicroscopic images of the oral cavity to distinguish pathological from healthy tissue. In: Bildverarbeitung für die Medizin 2015. Springer, Berlin, pp 479–485. https://doi.org/10.1007/978-3-662-46224-9_82
Kingma D, Ba J (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
Laemmel E, Genet M, Le Goualher G, Perchant A, Le Gargasson JF, Vicaut E (2004) Fibered confocal fluorescence microscopy (Cell-viZio) facilitates extended imaging in the field of microcirculation. A comparison with intravital microscopy. J Vasc Res 41(5):400–411. https://doi.org/10.1159/000081209
Macé V, Ahluwalia A, Coron E, Le Rhun M, Boureille A, Bossard C, Mosnier JF, Matysiak-Budnik T, Tarnawski AS (2015) Confocal laser endomicroscopy: a new gold standard for the assessment of mucosal healing in ulcerative colitis. J Gastroenterol Hepatol 30:85–92. https://doi.org/10.1111/jgh.12748
Maier AK, Schebesch F, Syben C, Würfl T, Steidl S, Choi JH, Fahrig R (2017) Precision learning: towards use of known operators in neural networks. arXiv preprint arXiv:1712.00374
Maier H, Dietz A, Gewelke U, Heller W, Weidauer H (1992) Tobacco and alcohol and the risk of head and neck cancer. Clin Investig 70(3–4):320–327. https://doi.org/10.1007/BF00184668
Muto M, Nakane M, Katada C, Sano Y, Ohtsu A, Esumi H, Ebihara S, Yoshida S (2004) Squamous cell carcinoma in situ at oropharyngeal and hypopharyngeal mucosal sites. Cancer 101(6):1375–1381. https://doi.org/10.1002/cncr.20482
Nachalon Y, Alkan U, Shvero J, Yaniv D, Shkedy Y, Limon D, Popovtzer A (2017) Assessment of laryngeal cancer in patients younger than 40 years. The Laryngoscope. https://doi.org/10.1002/lary.26951
Neumann H, Langner C, Neurath MF, Vieth M (2012) Confocal laser endomicroscopy for diagnosis of Barrett’s Esophagus. Front Oncol 2:42. https://doi.org/10.3389/fonc.2012.00042
Neumann H, Vieth M, Atreya R, Neurath MF, Mudter J (2011) Prospective evaluation of the learning curve of confocal laser endomicroscopy in patients with IBD. Histol Histopathol 26(7):867–872
Oetter N, Knipfer C, Rohde M, Wilmowsky C, Maier A, Brunner K, Adler W, Neukam FW, Neumann H, Stelzle F (2016) Development and validation of a classification and scoring system for the diagnosis of oral squamous cell carcinomas through confocal laser endomicroscopy. J Transl Med 14(1):1–11. https://doi.org/10.1186/s12967-016-0919-4
Oquab M, Bottou L, Laptev I, Sivic J (2015) Is object localization for free? Weakly-supervised learning with convolutional neural networks. In: Proceedings of IEEE conference on computer vision and pattern recognition. IEEE, pp 685–694. https://doi.org/10.1109/CVPR.2015.7298668
Parikh ND, Gibson J, Nagar A, Ahmed AA, Aslanian HR (2016) Confocal laser endomicroscopy features of sessile serrated adenomas/polyps. U Eur Gastroenterol J 4(4):599–603. https://doi.org/10.1177/2050640615621819
Pavlov V, Meyronet D, Meyer-Bisch V, Armoiry X, Pikul B, Dumot C, Beuriat P-A, Signorelli F, Guyotat J (2016) Intraoperative probe-based confocal laser endomicroscopy in surgery and stereotactic biopsy of low-grade and high-grade gliomas. Neurosurgery 79(4):604–612. https://doi.org/10.1227/NEU.0000000000001365
Robert Koch Institut. Zentrum für Krebsregisterdaten (2017) Krebs in Deutschland für 2013/2014, 11th edn. Robert Koch Institut, Berlin
Stoeve M, Aubreville M, Oetter N, Knipfer C, Neumann H, Stelzle F, Maier A (2018) Motion Artifact Detection in Confocal Laser Endomicroscopy Images. In: Bildverarbeitung für die Medizin. Springer, Berlin, pp 328–333. https://doi.org/10.1007/978-3-662-56537-7_85
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: The IEEE conference on computer vision and pattern recognition (CVPR). https://doi.org/10.1109/CVPR.2015.7298594
Vo K, Jaremenko Christian, Bohr C, Neumann H, Maier A (2017) Automatic classification and pathological staging of confocal laser endomicroscopic images of the vocal cords. In: Bildverarbeitung für die Medizin 2017. Springer, Berlin, pp 312–317. https://doi.org/10.1007/978-3-662-54345-0_70
Westra WH (2015) The pathology of HPV-related head and neck cancer: implications for the diagnostic pathologist. Semin Diagn Pathol 32(1):42–53. https://doi.org/10.1053/j.semdp.2015.02.023
Wiesner C, Jäger W, Salzer A, Biesterfeld S, Kiesslich R, Hampel C, Thüroff JW, Goetz M (2010) Confocal laser endomicroscopy for the diagnosis of urothelial bladder neoplasia: a technology of the future? BJU Int 107(3):399–403. https://doi.org/10.1111/j.1464-410X.2010.09540.x
Xing F, Xie Y, Su H, Liu F, Yang L (2017) Deep learning in microscopy image analysis: a survey. IEEE Trans Neural Netw Learn Syst PP(99):1–19. https://doi.org/10.1109/TNNLS.2017.2766168
Yang Y, Li T, Li W, Wu H, Fan W, Zhang W (2017) Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks. In: Medical image computing and computer-assisted intervention—MICCAI 2017. Springer, Cham, pp 533–540. https://doi.org/10.1007/978-3-319-66179-7_61
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All authors declare that they do not have conflicts of interest regarding the work covered by this manuscript.
Human and animal participants
All procedures involving human participants were in accordance with the 1964 Declaration of Helsinki and its later amendments.
Informed consent
The local ethics committee approved the studies (ethics committee of the University of Erlangen-Nürnberg; reference numbers 243 12 B and 60 14 B), and all patients gave their written informed consent.
Additional information
Marc Aubreville and Maike Stoeve have been contributing equally to the research in this paper.
Rights and permissions
About this article
Cite this article
Aubreville, M., Stoeve, M., Oetter, N. et al. Deep learning-based detection of motion artifacts in probe-based confocal laser endomicroscopy images. Int J CARS 14, 31–42 (2019). https://doi.org/10.1007/s11548-018-1836-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-018-1836-1