Abstract
Breast cancer diagnosis is a challenging task. Recently, the application of deep learning techniques to breast cancer diagnosis has become a popular trend. However, the effectiveness of deep neural networks is often limited by the lack of interpretability and the need for significant amount of manual annotations. To address these issues, we present a novel approach by leveraging both gaze data and multi-view data for mammogram classification. The gaze data of the radiologist serves as a low-cost and simple form of coarse annotation, which can provide rough localizations of lesions. We also develop a pyramid loss better fitting to the gaze-supervised process. Moreover, considering many studies overlooking interactive information relevant to diagnosis, we accordingly utilize transformer-based attention in our network to mutualize multi-view pathological information, and further employ a bidirectional fusion learning (BFL) to more effectively fuse multi-view information. Experimental results demonstrate that our proposed model significantly improves both mammogram classification performance and interpretability through incorporation of gaze data and cross-view interactive information.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adelson, E.H., Anderson, C.H., Bergen, J.R., Burt, P.J., Ogden, J.M.: Pyramid methods in image processing. RCA Eng. 29(6), 33–41 (1984)
Cheng, K., Ma, Y., Sun, B., Li, Y., Chen, X.: Depth estimation for colonoscopy images with self-supervised learning from videos. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12906, pp. 119–128. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87231-1_12
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Frazer, H.M., Qin, A.K., Pan, H., Brotchie, P.: Evaluation of deep learning-based artificial intelligence techniques for breast cancer detection on mammograms: results from a retrospective study using a breastscreen victoria dataset. J. Med. Imaging Radiat. Oncol. 65(5), 529–537 (2021)
Giaquinto, A.N., Miller, K.D., Tossas, K.Y., Winn, R.A., Jemal, A., Siegel, R.L.: Cancer statistics for African American/black people 2022. CA Cancer J. Clin. 72(3), 202–229 (2022)
Gutmann, M., Hyvärinen, A.: Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297–304. JMLR Workshop and Conference Proceedings (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Jiang, H., et al.: Eye tracking based deep learning analysis for the early detection of diabetic retinopathy: a pilot study. Available at SSRN 4247845 (2023)
Jørgensen, K.J., et al.: Breast-cancer screening-viewpoint of the iarc working group. New Engl. J. Med. 373, 1478 (2015)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kundel, H.L., Nodine, C.F., Krupinski, E.A., Mello-Thoms, C.: Using gaze-tracking data and mixture distribution analysis to support a holistic model for the detection of cancers on mammograms. Acad. Radiol. 15(7), 881–886 (2008)
Lee, R.S., Gimenez, F., Hoogi, A., Miyake, K.K., Gorovoy, M., Rubin, D.L.: A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data 4(1), 1–9 (2017)
Li, Z., et al.: Domain generalization for mammography detection via multi-style and multi-view contrastive learning. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12907, pp. 98–108. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87234-2_10
Lopez, E., Grassucci, E., Valleriani, M., Comminiello, D.: Multi-view breast cancer classification via hypercomplex neural networks. arXiv preprint arXiv:2204.05798 (2022)
Ma, C., et al.: Eye-gaze-guided vision transformer for rectifying shortcut learning. IEEE Trans. Med. Imaging (2023)
Moreira, I.C., Amaral, I., Domingues, I., Cardoso, A., Cardoso, M.J., Cardoso, J.S.: Inbreast: toward a full-field digital mammographic database. Acad. Radiol. 19(2), 236–248 (2012)
Oord, A.V.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
Ouyang, X., et al.: Learning hierarchical attention for weakly-supervised chest x-ray abnormality localization and diagnosis. IEEE Trans. Med. Imaging 40(10), 2698–2710 (2020)
Selvi, R.: Breast Diseases: Imaging and Clinical Management. Springer, Heidelberg (2014). https://doi.org/10.1007/978-81-322-2077-0
Shen, L., Margolies, L.R., Rothstein, J.H., Fluder, E., McBride, R., Sieh, W.: Deep learning to improve breast cancer detection on screening mammography. Sci. Rep. 9(1), 12495 (2019)
Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 1–11 (2017)
Voisin, S., Pinto, F., Xu, S., Morin-Ducote, G., Hudson, K., Tourassi, G.D.: Investigating the association of eye gaze pattern and diagnostic error in mammography. In: Medical Imaging 2013: Image Perception, Observer Performance, and Technology Assessment, vol. 8673, p. 867302. SPIE (2013)
Wang, S., Ouyang, X., Liu, T., Wang, Q., Shen, D.: Follow my eye: using gaze to supervise computer-aided diagnosis. IEEE Trans. Med. Imaging 41(7), 1688–1698 (2022)
Wu, C.C., Wolfe, J.M.: Eye movements in medical image perception: a selective review of past, present and future. Vision 3(2), 32 (2019)
Wu, N., et al.: Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans. Med. Imaging 39(4), 1184–1194 (2019)
Xian, J., Wang, Z., Cheng, K.-T., Yang, X.: Towards robust dual-view transformation via densifying sparse supervision for mammography lesion matching. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 355–365. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_34
Acknowledgements
This work was supported in part by The Key R &D Program of Guangdong Province, China (grant number 2021B0101420006), National Natural Science Foundation of China (grant numbers 62131015, 82272072), Science and Technology Commission of Shanghai Municipality (STCSM) (grant number 21010502600), and the CAAI-Huawei MindSpore Open Fund.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ji, C. et al. (2023). Mammo-Net: Integrating Gaze Supervision and Interactive Information in Multi-view Mammogram Classification. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14226. Springer, Cham. https://doi.org/10.1007/978-3-031-43990-2_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-43990-2_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43989-6
Online ISBN: 978-3-031-43990-2
eBook Packages: Computer ScienceComputer Science (R0)
