Skip to main content
Springer Nature Link
Log in

SATr: Slice Attention with Transformer for Universal Lesion Detection

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2022 (MICCAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13433))

Abstract

Universal Lesion Detection (ULD) in computed tomography plays an essential role in computer-aided diagnosis. Promising ULD results have been reported by multi-slice-input detection approaches which model 3D context from multiple adjacent CT slices, but such methods still experience difficulty in obtaining a global representation among different slices and within each individual slice since they only use convolution-based fusion operations. In this paper, we propose a novel Slice Attention Transformer (SATr) block which can be easily plugged into convolution-based ULD backbones to form hybrid network structures. Such newly formed hybrid backbones can better model long-distance feature dependency via the cascaded self-attention modules in the Transformer block while still holding a strong power of modeling local features with the convolutional operations in the original backbone. Experiments with five state-of-the-art methods show that the proposed SATr block can provide an almost free boost to lesion detection accuracy without extra hyperparameters or unique network designs. Code: https://github.com/MIRACLE-Center/A3D_SATr.

This research was supported in part by the Natural Science Foundation of China (grants 61732004 and 62176249).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Zlocha, M., Dou, Q., Glocker, B.: Improving RetinaNet for CT lesion detection with dense masks from weak RECIST labels. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 402–410. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_45

    Chapter  Google Scholar 

  2. Tao, Q., Ge, Z., Cai, J., Yin, J., See, S.: Improving deep lesion detection using 3D contextual and spatial attention. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 185–193. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_21

    Chapter  Google Scholar 

  3. Zhang, N., et al.: 3D anchor-free lesion detector on computed tomography scans. arXiv:1908.11324 (2019)

  4. Zhang, N., et al.: 3D aggregated faster R-CNN for general lesion detection. arXiv:2001.11071 (2020)

  5. Tang, Y., et al.: Uldor: a universal lesion detector for CT scans with pseudo masks and hard negative example mining. In: IEEE ISBI, pp. 833–836 (2019)

    Google Scholar 

  6. Yan, K., Bagheri, M., Summers, R.M.: 3D context enhanced region-based convolutional neural network for end-to-end lesion detection. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 511–519. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_58

    Chapter  Google Scholar 

  7. Li, Z., Zhang, S., Zhang, J., Huang, K., Wang, Y., Yu, Y.: MVP-Net: multi-view FPN with position-aware attention for deep universal lesion detection. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 13–21. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_2

    Chapter  Google Scholar 

  8. Yan, K., et al.: MULAN: multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 194–202. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_22

    Chapter  Google Scholar 

  9. Yang, J., et al.: AlignShift: bridging the gap of imaging thickness in 3D anisotropic volumes. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 562–572. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_55

    Chapter  Google Scholar 

  10. Cai, J., et al.: Deep volumetric universal lesion detection using light-weight pseudo 3D convolution and surface point regression. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 3–13. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_1

    Chapter  Google Scholar 

  11. Li, H., Han, H., Zhou, S.K.: Bounding maps for universal lesion detection. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 417–428. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_41

    Chapter  Google Scholar 

  12. Zhang, S., et al.: Revisiting 3D context modeling with supervised pre-training for universal lesion detection in CT slices. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 542–551. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_53

    Chapter  Google Scholar 

  13. Yan, K., et al.: Learning from multiple datasets with heterogeneous and partial labels for universal lesion detection in CT. IEEE Trans. Med. Imaging 40, 2759–2770 (2020)

    Article  Google Scholar 

  14. Cai, J., et al.: Deep lesion tracker: monitoring lesions in 4D longitudinal imaging studies. In: IEEE CVPR, pp. 15159–15169 (2021)

    Google Scholar 

  15. Tang, Y., et al.: Weakly-supervised universal lesion segmentation with regional level set loss. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 515–525. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_48

    Chapter  Google Scholar 

  16. Yang, J., He, Y., Kuang, K., Lin, Z., Pfister, H., Ni, B.: Asymmetric 3D context fusion for universal lesion detection. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 571–580. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_55

    Chapter  Google Scholar 

  17. Li, H., Chen, L., Han, H., Chi, Y., Zhou, S.K.: Conditional training with bounding map for universal lesion detection. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 141–152. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_14

    Chapter  Google Scholar 

  18. Lyu, F., Yang, B., Ma, A.J., Yuen, P.C.: A segmentation-assisted model for universal lesion detection with partial labels. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 117–127. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_12

    Chapter  Google Scholar 

  19. Boot, T., Irshad, H.: Diagnostic assessment of deep learning algorithms for detection and segmentation of lesion in mammographic images. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 56–65. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_6

    Chapter  Google Scholar 

  20. Yu, X., et al.: Deep attentive panoptic model for prostate cancer detection using biparametric MRI scans. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 594–604. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_58

    Chapter  Google Scholar 

  21. Ren, Y., et al.: Retina-match: ipsilateral mammography lesion matching in a single shot detection pipeline. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 345–354. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_33

    Chapter  Google Scholar 

  22. Baumgartner, M., Jäger, P.F., Isensee, F., Maier-Hein, K.H.: nnDetection: a self-configuring method for medical object detection. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 530–539. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_51

    Chapter  Google Scholar 

  23. Shahroudnejad, A., et al.: TUN-Det: a novel network for thyroid ultrasound nodule detection. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12901, pp. 656–667. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87193-2_62

    Chapter  Google Scholar 

  24. Luo, L., Chen, H., Zhou, Y., Lin, H., Heng, P.-A.: OXnet: deep omni-supervised thoracic disease detection from chest X-rays. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12902, pp. 537–548. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87196-3_50

    Chapter  Google Scholar 

  25. Chen, J., Zhang, Y., Wang, J., Zhou, X., He, Y., Zhang, T.: EllipseNet: anchor-free ellipse detection for automatic cardiac biometrics in fetal echocardiography. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12907, pp. 218–227. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_21

    Chapter  Google Scholar 

  26. Yang, H.-H., et al.: Leveraging auxiliary information from EMR for weakly supervised pulmonary nodule detection. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12907, pp. 251–261. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_24

    Chapter  Google Scholar 

  27. Lin, C., Wu, H., Wen, Z., Qin, J.: Automated Malaria cells detection from blood smears under severe class imbalance via importance-aware balanced group softmax. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 455–465. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_44

    Chapter  Google Scholar 

  28. Zhao, Z., Pang, F., Liu, Z., Ye, C.: Positive-unlabeled learning for cell detection in histopathology images with incomplete annotations. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 509–518. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_49

    Chapter  Google Scholar 

  29. Kevin Zhou, S., et al.: A review of deep learning in medical imaging: imaging traits, technology trends, case studies with progress highlights, and future promises (2021)

    Google Scholar 

  30. Kevin Zhou, S., Rueckert, D., Fichtinger, G.: Handbook of Medical Image Computing and Computer Assisted Intervention. Academic Press (2019)

    Google Scholar 

  31. Huang, G., et al.: Densely connected convolutional networks. In: IEEE CVPR, pp. 4700–4708 (2017)

    Google Scholar 

  32. Lin, T., et al.: Feature pyramid networks for object detection. In: IEEE CVPR, pp. 2117–2125 (2017)

    Google Scholar 

  33. Peng, Z., et al.: Conformer: local features coupling global representations for visual recognition. In: IEEE ICCV, pp. 367–376 (2021)

    Google Scholar 

  34. Xu, Y., et al.: ViTAE: vision transformer advanced by exploring intrinsic inductive bias. In: NeurlIPS, vol. 34 (2021)

    Google Scholar 

  35. Mao, M., et al.: Dual-stream network for visual recognition. In: NeurlIPS, vol. 34 (2021)

    Google Scholar 

  36. Yan, K., et al.: Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings in a diverse large-scale lesion database. In: IEEE CVPR, pp. 9261–9270 (2018)

    Google Scholar 

  37. Zhu, X., et al.: Deformable detr: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020)

  38. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  39. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS, pp. 91–99 (2015)

    Google Scholar 

  40. Gildenblat, J., et al.: Pytorch library for cam methods (2021). https://github.com/jacobgil/pytorch-grad-cam

  41. Muhammad, M.B., et al. Eigen-CAM: class activation map using principal components. In: IEEE IJCNN, pp. 1–7 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Biomedical Engineering and Suzhou Institute for Advanced Research Center for Medical Imaging, Robotics, and Analytic Computing and Learning (MIRACLE), University of Science and Technology of China, Suzhou, 215123, China

    Han Li & S. Kevin Zhou

  2. Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing, 100190, China

    Han Li, Long Chen, Hu Han & S. Kevin Zhou

  3. School of Computer Science and Technology, University of the Chinese Academy of Science, Beijing, China

    Long Chen

Authors
  1. Han Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Long Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hu Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. S. Kevin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Hu Han or S. Kevin Zhou .

Editor information

Editors and Affiliations

  1. Rochester Institute of Technology, Rochester, NY, USA

    Linwei Wang

  2. Chinese University of Hong Kong, Hong Kong, Hong Kong

    Qi Dou

  3. University of Virginia, Charlottesville, VA, USA

    P. Thomas Fletcher

  4. National Center for Tumor Diseases (NCT/UCC), Dresden, Germany

    Stefanie Speidel

  5. Case Western Reserve University, Cleveland, OH, USA

    Shuo Li

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 458 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, H., Chen, L., Han, H., Kevin Zhou, S. (2022). SATr: Slice Attention with Transformer for Universal Lesion Detection. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13433. Springer, Cham. https://doi.org/10.1007/978-3-031-16437-8_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16437-8_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16436-1

  • Online ISBN: 978-3-031-16437-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships