Skip to main content

Advertisement

SpringerLink
Log in

A data-efficient visual analytics method for human-centered diagnostic systems to endoscopic ultrasonography

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Endoscopic ultrasound (EUS) has emerged as a pivotal tool for the screening and diagnosis of submucosal tumors (SMTs). However, the inherently low-quality and highly variable image content presents substantial obstacles to the automation of SMT diagnosis. Deep learning, with its adaptive feature extraction capabilities, offers a potential solution, yet its implementation often requires a vast quantity of high-quality data - a challenging prerequisite in clinical settings. To address this conundrum, this paper proposes a novel data-efficient visual analytics method that integrates human feedback into the model lifecycle, thereby augmenting the practical utility of data. The methodology leverages a two-stage deep learning algorithm, which encompasses self-supervised pre-training and an attention-based network. Comprehensive experimental validation reveals that the proposed approach facilitates the model in deciphering the hierarchical structure information within high-noise EUS images. Moreover, when allied with human-machine interaction, it enhances data utilization, thereby elevating the accuracy and reliability of diagnostic outcomes. The code is available at https://github.com/Zehebi29/LA-RANet.git.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Nishida T, Kawai N, Yamaguchi S, Nishida Y (2013) Submucosal tumors: comprehensive guide for the diagnosis and therapy of gastrointestinal submucosal tumors. Digest Endosc 25:479–489. https://doi.org/10.1111/den.12149

    Article  Google Scholar 

  2. Landi B, Palazzo L (2009) The role of endosonography in submucosal tumours. Best Pract Res Clin Gastroenterol 23:679–701. https://doi.org/10.1016/j.bpg.2009.05.009

    Article  Google Scholar 

  3. Chak A (2002) Eus in submucosal tumors. IEEE Trans Med Imaging 56:43–48. https://doi.org/10.1067/mge.2002.127700

    Article  Google Scholar 

  4. Chen X, Hu Y, Zhang Z, Wang B, Zhang L, Shi F, Chen X, Jiang X (2019) A graph-based approach to automated eus image layer segmentation and abnormal region detection. Neurocomputing 336:79–91. https://doi.org/10.1016/j.neucom.2018.03.083

    Article  Google Scholar 

  5. Shen Y, Ke, J (2021) Sampling based tumor recognition in whole-slide histology image with deep learning approaches. IEEE/ACM Trans Comput Biol Bioinform

  6. Zhang J, Zhu L, Yao L, Ding X, Chen D, Wu H, Lu Z, Zhou W, Zhang L, An P et al (2020) Deep learning-based pancreas segmentation and station recognition system in eus: development and validation of a useful training tool (with video). Gastrointest Endosc 92(4):874–885

    Article  Google Scholar 

  7. Iwasa Y, Iwashita T, Takeuchi Y, Ichikawa H, Mita N, Uemura S, Shimizu M, Kuo Y-T, Wang H-P, Hara T (2021) Automatic segmentation of pancreatic tumors using deep learning on a video image of contrast-enhanced endoscopic ultrasound. J Clin Med 10(16):3589

    Article  Google Scholar 

  8. Liu E, Bhutani MS, Sun S (2021) Artificial intelligence: The new wave of innovation in eus. Endosc Ultrasound 10(2):79

    Article  Google Scholar 

  9. Liu Q, Yu L, Luo L, Dou Q, Heng PA (2020) Semi-supervised medical image classification with relation-driven self-ensembling model. IEEE Trans Med Imaging 39(11):3429–3440

    Article  Google Scholar 

  10. Taleb A, Lippert C, Klein T, Nabi M (2021) Multimodal self-supervised learning for medical image analysis. In: Inf Process Med Imaging, pp 661–673. Springer

  11. Li X, Jia M, Islam MT, Yu L, Xing L (2020) Self-supervised feature learning via exploiting multi-modal data for retinal disease diagnosis. IEEE Trans Med Imaging 39(12):4023–4033

    Article  Google Scholar 

  12. Mahapatra D, Poellinger A, Shao L, Reyes M (2021) Interpretability-driven sample selection using self supervised learning for disease classification and segmentation. IEEE Trans Med Imaging

  13. Zhang Y, Li M, Ji Z, Fan W, Yuan S, Liu Q, Chen Q (2021) Twin self-supervision based semi-supervised learning (ts-ssl): Retinal anomaly classification in sd-oct images. Neurocomput 462:491–505

    Article  Google Scholar 

  14. Lee JD, Lei Q, Saunshi N, Zhuo J (2020) Predicting what you already know helps: Provable self-supervised learning. arXiv:2008.01064

  15. Li X, Wang W, Hu X, Yang J (2019) Selective kernel networks. In: Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR), pp 510–519

  16. Hu Y, Wen G, Luo M, Yang P, Dai D, Yu Z, Wang C, Hall W (2021) Fully-channel regional attention network for disease-location recognition with tongue images. Artif Intell Med, p 102110

  17. Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: Convolutional block attention module. In: Proc Eur Conf Comput Vis, pp 3–19

  18. Cheng J, Tian S, Yu L, Lu H, Lv X (2020) Fully convolutional attention network for biomedical image segmentation. Artif Intell Med 107:101899

    Article  Google Scholar 

  19. Zhao H, Jia J, Koltun V (2020) Exploring self-attention for image recognition. In: Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR), pp 10076–10085

  20. Kuwahara T, Hara K, Mizuno N, Okuno N, Matsumoto S, Obata M, Kurita Y, Koda H, Toriyama K, Onishi S, et al (2019) Usefulness of deep learning analysis for the diagnosis of malignancy in intraductal papillary mucinous neoplasms of the pancreas. Clin Tansl Gastroen 10(5)

  21. Chen L, Bentley P, Mori K, Misawa K, Fujiwara M, Rueckert D (2019) Self-supervised learning for medical image analysis using image context restoration. Med Image Anal 58:101539

    Article  Google Scholar 

  22. Liu X, Sinha A, Ishii M, Hager GD, Reiter A, Taylor RH, Unberath M (2019) Dense depth estimation in monocular endoscopy with self-supervised learning methods. IEEE Trans Med Imaging 39(5):1438–1447

    Article  Google Scholar 

  23. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: Feature learning by inpainting. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 2536–2544

  24. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 4681–4690

  25. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 2223–2232

  26. Zhang R, Isola P, Efros AA (2016) Colorful image colorization. In: Comput Vis ECCV, pp 649–666. Springer

  27. Kim D, Cho D, Yoo D, Kweon IS (2018) Learning image representations by completing damaged jigsaw puzzles. In: 2018 IEEE Winter Conf on Applications of Comput Vis (WACV), pp 793–802. IEEE

  28. Komodakis N, Gidaris S (2018) Unsupervised representation learning by predicting image rotations. In: International Conference on Learning Representations (ICLR)

  29. Ren, Z, Lee YJ (2018) Cross-domain self-supervised multi-task feature learning using synthetic imagery. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 762–771

  30. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 3156–3164

  31. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 7132–7141

  32. Ma J, Zhang H, Yi P, Wang Z (2019) Scscn: A separated channel-spatial convolution net with attention for single-view reconstruction. IEEE T Ind Electron 67(10):8649–8658

    Article  Google Scholar 

  33. Huang C, Lan Y, Xu G, Zhai X, Wu J, Lin F, Zeng N, Hong Q, Ng E, Peng Y et al (2020) A deep segmentation network of multi-scale feature fusion based on attention mechanism for ivoct lumen contour. IEEE/ACM Trans Comput Biol Bioinform 18(1):62–69

    Article  Google Scholar 

  34. Tong H, Fang Z, Wei Z, Cai Q, Gao Y (2021) Sat-net: a side attention network for retinal image segmentation. Appl Intell 51(7):5146–5156

    Article  Google Scholar 

  35. Rao A, Park J, Woo S, Lee J-Y, Aalami O (2021) Studying the effects of self-attention for medical image analysis. In: Proc of the IEEE/CVF International Conf on Comput Vis, pp 3416–3425

  36. Cao Y, Xu J, Lin S, Wei F, Hu H (2019) Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In: Proc of the IEEE/CVF International Conf on Comput Vis Workshops, pp 0–0

  37. Buades A, Coll B, Morel J-M (2005) A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol 2, pp 60–65. IEEE

  38. Zuiderveld K (1994) Contrast limited adaptive histogram equalization. Graphics gems, pp 474–485

  39. Bradski G (2000) The OpenCV Library. Dr. Dobb’s Journal of Software Tools

  40. Suzuki S et al (1985) Topological structural analysis of digitized binary images by border following. Comput Gr Image Process 30(1):32–46

    Article  Google Scholar 

  41. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  42. Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter SL, McKeown A, Yang G, Wu X, Yan F et al (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5):1122–1131

    Article  Google Scholar 

  43. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 770–778

  44. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International conference on computer vision, pp 10012–10022

  45. Mehta S, Rastegari M (2021) Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv:2110.02178

  46. He J, Chen J-N, Liu S, Kortylewski A, Yang C, Bai Y, Wang C (2022) Transfg: A transformer architecture for fine-grained recognition. Proceedings of the AAAI Conference on artificial intelligence 36:852–860

    Article  Google Scholar 

  47. Liu Z, Mao H, Wu C-Y, Feichtenhofer C, Darrell T, Xie S (2022) A convnet for the 2020s. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 11976–11986

  48. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: Comput Vis ECCV, pp 818–833. Springer

  49. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 2921–2929

  50. Wang H, Wang Z, Du M, Yang F, Zhang Z, Ding S, Mardziel P, Hu X (2020) Score-cam: Score-weighted visual explanations for convolutional neural networks. In: Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR), pp 24–25

Download references

Acknowledgements

This study was supported by the action plan for the scientific and technological innovation Program from the Shanghai Science and Technology Committee (Nos.19411951500)

Author information

Authors and Affiliations

  1. College of Mechanical Engineering, Dong Hua University, Shanghai, 201620, China

    Hangbin Zheng & Jinsong Bao

  2. Endoscopy Center, Shanghai Jiaotong University Affiliated Sixth People’s Hospital, Shanghai, 200233, China

    Zhixia Dong & Xinjian Wan

Authors
  1. Hangbin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinsong Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhixia Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinjian Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinsong Bao.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, H., Bao, J., Dong, Z. et al. A data-efficient visual analytics method for human-centered diagnostic systems to endoscopic ultrasonography. Appl Intell 53, 30822–30842 (2023). https://doi.org/10.1007/s10489-023-05088-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05088-0

Keywords

Search

Navigation