Abstract
Most of the existing multi-view mammographic image analysis methods adopt a simple fusion strategy: features concatenation, which is widely used in many features fusion methods. However, concatenation based methods can’t extract cross view information very effectively because different views are likely to be unaligned. Recently, many researchers have attempted to introduce attention mechanism related methods into the field of multi-view mammography analysis. But these attention mechanism based methods still partly rely on convolution, so they can’t take full advantages of attention mechanism. To take full advantage of multi-view information, we propose a novel pure transformer based multi-view network to solve the question of mammographic image classification. In our primary network, we use a transformer based backbone network to extract image features, a “cross view attention block” structure to fuse multi-view information, and a “classification token” to gather all useful information to make the final prediction. Besides, we compare the performance when fusing multi-view information at different stages of the backbone network using a novel designed “(shifted) window based cross view attention block” structure and compare the results when fusing different views’ information. The results on DDSM dataset show that our networks can effectively use multi-view information to make judgments and outperform the concatenation and convolution based methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sun, L., Wang, J., Hu, Z., Xu, Y., Cui, Z.: Multi-view convolutional neural networks for mammographic image classification. IEEE Access 7, 126273–126282 (2019)
Nasir Khan, H., Shahid, A.R., Raza, B., Dar, A.H., Alquhayz, H.: Multi-view feature fusion based four views model for mammogram classification using convolutional neural network. IEEE Access 7, 165724–165733 (2019)
Wu, N., et al.: Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Trans. Med. Imaging 39, 1184–1194 (2020)
Li, C., et al.: Multi-view mammographic density classification by dilated and attention-guided residual learning. IEEE/ACM Trans. Comput. Biol. Bioinf. 18, 1003–1013 (2021)
Zhao, X., Yu, L., Wang, X.: Cross-view attention network for breast cancer screening from multi-view mammograms. In: 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2020, pp. 1050–1054. IEEE, Barcelona (2020)
Hu, H., Gu, J., Zhang, Z., Dai, J., Wei, Y.: Relation networks for object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3588–3597. IEEE, Salt Lake City (2018)
Ma, J., Li, X., Li, H., Wang, R., Menze, B., Zheng, W.-S.: Cross-view relation networks for mammogram mass detection. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 8632–8638. IEEE, Milan (2021)
Yang, Z., et al.: MommiNet-v2: mammographic multi-view mass identification networks. Med. Image Anal. 73, 102204 (2021)
van Tulder, G., Tong, Y., Marchiori, E.: Multi-view analysis of unregistered medical images using cross-view transformers. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 104–113. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87199-4_10
Liu, Y., et al.: Compare and contrast: detecting mammographic soft-tissue lesions with C2 -Net. Med. Image Anal. 71, 101999 (2021)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30, pp. 5998–6008 (2017)
Dosovitskiy, A., et al.: An image is worth 16 × 16 words: transformers for image recognition at scale. In: ICLR (2021)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: ICCV (2021)
Tsai, Y.-H.H., Bai, S., Liang, P.P., Kolter, J.Z., Morency, L.-P., Salakhutdinov, R.: Multimodal transformer for unaligned multimodal language sequences. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6558–6569. Association for Computational Linguistics, Florence (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE, Las Vegas (2016)
Heath, M., Bowyer, K., Kopans, D., Kegelmeyer, P., Moore, R., Chang, K.: Current status of the digital database for screening mammography. In: Karssemeijer, N., Thijssen, M., Hendriks, J., van Erning, L. (eds.) Digital Mammography. Computational Imaging and Vision, vol. 13, pp. 457–460. Springer, Dordrecht (1998). https://doi.org/10.1007/978-94-011-5318-8_75
Yan, Y., Conze, P.-H., Lamard, M., Quellec, G., Cochener, B., Coatrieux, G.: Towards improved breast mass detection using dual-view mammogram matching. Med. Image Anal. 71, 102083 (2021)
Acknowledgement
This research is supported by Zhengzhou collaborative innovation major special project (20XTZX11020).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Sun, Z., Jiang, H., Ma, L., Yu, Z., Xu, H. (2022). Transformer Based Multi-view Network for Mammographic Image Classification. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13433. Springer, Cham. https://doi.org/10.1007/978-3-031-16437-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-16437-8_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16436-1
Online ISBN: 978-3-031-16437-8
eBook Packages: Computer ScienceComputer Science (R0)
