Skip to main content
Springer Nature Link
Log in

Robust endocytoscopic image classification based on higher-order symmetric tensor analysis and multi-scale topological statistics

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

An endocytoscope is a new type of endoscope that enables users to perform conventional endoscopic observation and ultramagnified observation at the cell level. Although endocytoscopy is expected to improve the cost-effectiveness of colonoscopy, endocytoscopic image diagnosis requires much knowledge and high-level experience for physicians. To circumvent this difficulty, we developed a robust endocytoscopic (EC) image classification method for the construction of a computer-aided diagnosis (CAD) system, since real-time CAD can resolve accuracy issues and reduce interobserver variability.

Method

We propose a novel feature extraction method by introducing higher-order symmetric tensor analysis to the computation of multi-scale topological statistics on an image, and we integrate this feature extraction with EC image classification. We experimentally evaluate the classification accuracy of our proposed method by comparing it with three deep learning methods. We conducted this comparison by using our large-scale multi-hospital dataset of about 55,000 images of over 3800 patients.

Results

Our proposed method achieved an average 90% classification accuracy for all the images in four hospitals even though the best deep learning method achieved 95% classification accuracy for images in only one hospital. In the case with a rejection option, the proposed method achieved expert-level accurate classification. These results demonstrate the robustness of our proposed method against pit pattern variations, including differences of colours, contrasts, shapes, and hospitals.

Conclusions

We developed a robust EC image classification method with novel feature extraction. This method is useful for the construction of a practical CAD system, since it has sufficient generalisation ability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Mori Y, Kudo SE, Ikehara N, Wada Y, Kutsukawa M, Misawa M, Kudo T, Kobayashi Y, Miyachi H, Yamamura F, Ohtsuka K, Inoue H, Hamatani S (2013) Comprehensive diagnostic ability of endocytoscopy compared with biopsy for colorectal neoplasms: a prospective randomized noninferiority trial. Endoscopy 45(2):98–105

    Article  CAS  Google Scholar 

  2. Mori Y, Kudo SE, Chiu PW, Singh R, Misawa M, Wakamura K, Kudo T, Hayashi T, katagiri A, Miyachi H, Ishida F, Maeda Y, Inoue H, Nimura Y, Oda M, Mori K (2016) Impact of an automated system for endocytoscopic diagnosis of small colorectal lesions: an international web-based study. Endoscopy 48(12):1110–1118

    Article  Google Scholar 

  3. Fu JJ, Yu YW, Lin HM, Chai JW, Chen CC (2014) Feature extraction and pattern classification of colorectal polyps in colonoscopic imaging. Comput Med Imag Graph 38(4):267–275

    Article  Google Scholar 

  4. Mesejo P, Pizarro D, Abergel A, Rouquette O, Beorchia S, Poincloux L, Bartoli A (2016) Computer-aided classification of gastrointestinal lesions in regular colonoscopy. IEEE Trans Med Imag 35(9):2051–2063

    Article  Google Scholar 

  5. Häfner M, Liedlgruber M, Uhl A, Vécsei A, Wrba F (2012) Color treatment in endoscopic image classification using multi-scale local color vector patterns. Med Imag Anal 16(1):75–86

    Article  Google Scholar 

  6. Tamaki T, Yoshimura J, Kawakami M, Raytchev B, Kaneda K, Yoshida S, Takemura Y, Onji K, Miyaki R, Tanaka S (2013) Computer-aided colorectal tumor classification in NBI endoscopy using local features. Med Imag Anal 17(1):78–100

    Article  Google Scholar 

  7. Wimmer G, Tamaki T, Tischendorf JJ, Häfner M, Yoshida S, Tanaka S, Uhl A (2016) Directional wavelet based features for colonic polyp classification. Med Imag Anal 31:16–36

    Article  Google Scholar 

  8. Tamaki T, Sonoyama S, Hirakawa T, Raytchev B, Kaneda K, Koide T, Yoshida S, Mieno Hiroshi, Tanaka S (2016) Computer-aided colorectal tumor classification in NBI endoscopy using CNN features. In: Proc Korea–Japan joint workshop on Frontiers of computer vision, pp 61–65

  9. Itoh H, Mori Y, Misawa M, Oda M, Kudo SE, Mori K (2018) Cascade classification of endocytoscopic images of colorectal lesions for automated pathological diagnosis. Proc SPIE Med Imag 10575:269–274

    Google Scholar 

  10. Itoh H, Lu Z, Mori Y, Misawa M, Oda M, Kudo SE, Mori K (2018) Discriminative feature selection by optimal manifold search for neoplastic image recognition. Proc ECCV Workshops 4:534–549

    Google Scholar 

  11. Kominami Y, Yoshida S, Tanaka S, Sanomura Y, Hirakawa T, Raytchev B, Tamaki T, Koide T, Kaneda K, Chayama K (2016) Computer-aided diagnosis of colorectal polyp histology by using a real-time image recognition system and narrow-band imaging magnifying colonoscopy. Gastrointest Endosc 83:643–649

    Article  Google Scholar 

  12. Schultz T, Weickert J, Seidel HP (2009) A higher-order structure tensor. In: Visualization and processing of tensor fields. Springer, Berlin

  13. Schultz T (2011) Topological features in 2D symmetric higher-order tensor fields. Comput Gr Forum 30(3):841–850

    Article  Google Scholar 

  14. Haralick RM, Shanmugam K, Dinstein I (1973) Textural features for image classification. IEEE Trans Syst Man Cybern SMC 3(6):610–621

    Article  Google Scholar 

  15. Bishop CM (2011) Pattern recognition and machine learning, 2nd edn. Springer, Berlin

    Google Scholar 

  16. Vapnik VN (1998) Statistical learning theory. Wiley, Hoboken

    Google Scholar 

  17. Platt JC (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifier, pp 61–74

  18. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Proc ECCV, pp 818–833

  19. Maas AL, Hannun AY, Ng AY (2013) Rectifier nonlinearities improve neural network acoustic models. Proc Int Conf Mach Learn 30(1):3

    Google Scholar 

  20. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. Proc Int Conf Mach Learn 37:448–456

    Google Scholar 

  21. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    Google Scholar 

  22. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc CVPR, pp 770–778

  23. Veit A, Wilber M, Belongie S (2016) Residual networks behave like ensembles of relatively shallow networks. In: Proc NIPS, pp 550–558

  24. Wang J, Li X, Ling CX (2018) Pelee: a real-time object detection system on mobile devices. In: Proc NIPS, pp 1963–1972

  25. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proc ICCV, pp 1026–1034

  26. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Proc ICLR, pp 1–13

  27. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Li FF (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  Google Scholar 

  28. Mori Y, Kudo SE, Misawa M, Saito Y, Ikematsu H, Hotta K, Ohtsuka K, Urushibara F, Kataoka S, Ogawa Y, Maeda Y, Takeda K, Nakamura H, Ichimasa K, Kudo T, Hayashi T, Wakamura K, Ishida F, Inoue H, Itoh H, Oda M, Mori K (2018) Real-time use of artificial intelligence in identification of diminutive polyps during colonoscopy: a prospective study. Ann Int Med 169(6):357–366

    Article  Google Scholar 

Download references

Acknowledgements

This study was funded by Grants from AMED (19hs0110006h0003), JSPS MEXT KAKENHI (26108006, 17H00867, 17K20099, 19K08403), and the JSPS Bilateral Joint Research Project.

Author information

Authors and Affiliations

  1. Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan

    Hayato Itoh, Yuichiro Hayashi, Masahiro Oda & Kensaku Mori

  2. Information Strategy Office, Information and Communications, Nagoya University, Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan

    Yukitaka Nimura

  3. Digestive Disease Center, Showa University Northern Yokohama Hospital, Chigasaki-chuo 35-1, Tsuduki-ku, Yokohama, 224-8503, Japan

    Yuichi Mori, Masashi Misawa & Shin-Ei Kudo

  4. Division of Endoscopy, Shizuoka Cancer Center, Shimonagakubo 1007, Nagaizumi-cho, Sunto-gun, Shizuoka, 411-8777, Japan

    Kinichi Hotta

  5. Department of Gastroenterology and Hepatology, Tokyo Medical and Dental University, Yushima 1-5-45, Bunkyo-ku, Tokyo, 113-8510, Japan

    Kazuo Ohtsuka

  6. Department of Gastroenterology, Cancer Institute Hospital of Japanese Foundation for Cancer Research, Ariake 3-8-31, Koto-ku, Tokyo, 135-8550, Japan

    Shoichi Saito

  7. Endoscopy Division, National Cancer Center Hospital, Tsukiji 5-1-1, Chuo-ku, Tokyo, 104-0045, Japan

    Yutaka Saito

  8. Department of Gastroenterology and Endoscopy, National Cancer Center Hospital East, Kashiwanoha 6-5-1, Kashiwa, 277-8577, Japan

    Hiroaki Ikematsu

Authors
  1. Hayato Itoh
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yukitaka Nimura
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yuichi Mori
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Masashi Misawa
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Shin-Ei Kudo
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Kinichi Hotta
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Kazuo Ohtsuka
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Shoichi Saito
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Yutaka Saito
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Hiroaki Ikematsu
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Yuichiro Hayashi
    View author publications

    You can also search for this author inPubMed Google Scholar

  12. Masahiro Oda
    View author publications

    You can also search for this author inPubMed Google Scholar

  13. Kensaku Mori
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Hayato Itoh.

Ethics declarations

Conflict of interest

Kudo SE, Misawa M, and Mori Y received lecture fees from Olympus. Ohtsuka K reports personal fees and nonfinancial support from Olympus outside of this work. Mori K is supported by Cybernet Systems and Olympus (Research Grant) in this work, and by NTT outside of the submitted work. The other authors have no conflicts of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical committee of Nagoya University (No. 351, 357) and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained via an opt-out procedure from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Itoh, H., Nimura, Y., Mori, Y. et al. Robust endocytoscopic image classification based on higher-order symmetric tensor analysis and multi-scale topological statistics. Int J CARS 15, 2049–2059 (2020). https://doi.org/10.1007/s11548-020-02255-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02255-3

Keywords