Skip to main content
SpringerLink
Log in

Multi-branch cross attention model for prediction of KRAS mutation in rectal cancer with t2-weighted MRI

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The accurate identification of KRAS mutation status on medical images is critical for doctors to specify treatment options for patients with rectal cancer. Deep learning methods have recently been successfully introduced to medical diagnosis and treatment problems, although substantial challenges remain in the computer-aided diagnosis (CAD) due to the lack of large training datasets. In this paper, we propose a multi-branch cross attention model (MBCAM) to separate KRAS mutation cases from wild type cases using limited T2-weighted MRI data. Our model is built on multiple different branches generated based on our existing MRI data, which can take full advantage of the information contained in small data sets. The cross attention block (CA block) is proposed to fuse formerly independent branches to ensure that the model can learn as many common features as possible for preventing the overfitting of the model due to the limited dataset. The inter-branch loss is proposed to constrain the learning range of the model, confirming that the model can learn more general features from multi-branch data. We tested our method on the collected dataset and compared it to four previous works and five popular deep learning models using transfer learning. Our result shows that the MBCAM achieved an accuracy of 88.92% for the prediction of KRAS mutations with an AUC of 95.75%. These results are a significant improvement over those existing methods (p < 0.05).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. https://docs.opencv.org/3.2.0/d4/d86/group__imgproc__filter.html#gacea54f142e81b6758cb6f375ce782c8d

  2. https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI.

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pp 265–283

  2. Afshar P, Mohammadi A, Plataniotis KN, Oikonomou A, Benali H (2018) From hand-crafted to deep learning-based cancer radiomics: challenges and opportunities. arXiv:https://arxiv.org/abs/1808.07954

  3. Altaf F, Islam S, Akhtar N, Janjua NK (2019) Going deep in medical image analysis: concepts, methods, challenges and future directions. arXiv:https://arxiv.org/abs/1902.05655

  4. Armaghany T, Wilson JD, Chu Q, Mills G (2012) Genetic alterations in colorectal cancer. Gastrointestinal Cancer Research: GCR 5(1):19

    Google Scholar 

  5. Calimeri F, Marzullo A, Stamile C, Terracina G (2017) Biomedical data augmentation using generative adversarial neural networks. In: International conference on artificial neural networks. Springer, pp 626–634

  6. Chai Y, Liu H, Xu J (2018) Glaucoma diagnosis based on both hidden features and domain knowledge through deep learning models. Knowl-Based Syst 161:147–156

    Article  Google Scholar 

  7. Chen LC, Yang Y, Wang J, Xu W, Yuille AL (2016) Attention to scale: scale-aware semantic image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3640–3649

  8. Chollet F, et al. (2015) Keras

  9. Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyö D., Moreira AL, Razavian N, Tsirigos A (2018) Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nat Med 24(10):1559

    Article  Google Scholar 

  10. Cui Y, Cui X, Yang X, Zhuo Z, Du X, Xin L, Yang Z, Cheng X (2019) Diffusion kurtosis imaging-derived histogram metrics for prediction of kras mutation in rectal adenocarcinoma: preliminary findings. Journal of Magnetic Resonance Imaging

  11. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 248–255

  12. Dong N, Kampffmeyer M, Liang X, Wang Z, Dai W, Xing E (2018) Unsupervised domain adaptation for automatic estimation of cardiothoracic ratio. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 544–552

  13. Frid-Adar M, Diamant I, Klang E, Amitai M, Goldberger J, Greenspan H (2018) Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification. Neurocomputing 321:321–331

    Article  Google Scholar 

  14. Gevaert O, Echegaray S, Khuong A, Hoang CD, Shrager JB, Jensen KC, Berry GJ, Guo HH, Lau C, Plevritis SK et al (2017) Predictive radiogenomics modeling of egfr mutation status in lung cancer. Sci Rep 7(41):674

    Google Scholar 

  15. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  16. Horvat N, Veeraraghavan H, Pelossof RA, Fernandes MC, Arora A, Khan M, Marco M, Cheng CT, Gonen M, Pernicka JSG, et al (2019) Radiogenomics of rectal adenocarcinoma in the era of precision medicine: a pilot study of associations between qualitative and quantitative mri imaging features and genetic mutations. Eur J Radiol 113:174–181

    Article  Google Scholar 

  17. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:https://arxiv.org/abs/1704.04861

  18. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  19. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  20. Huang J, Ling CX (2005) Using auc and accuracy in evaluating learning algorithms. IEEE Trans Knowl Data Eng 17(3):299– 310

    Article  Google Scholar 

  21. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:https://arxiv.org/abs/1502.03167

  22. Jia S, Chen D, Chen H (2019) Instance-level meta normalization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4865–4873

  23. Kamper H, Wang W, Livescu K (2016) Deep convolutional acoustic word embeddings using word-pair side information. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 4950–4954

  24. Kim B, Kim H, Kim K, Kim S, Kim J (2019) Learning not to learn: training deep neural networks with biased data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9012–9020

  25. Koch G, Zemel R, Salakhutdinov R (2015) Siamese neural networks for one-shot image recognition. In: ICML deep learning workshop, vol 2

  26. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  27. Labianca R, Beretta GD, Kildani B, Milesi L, Merlin F, Mosconi S, Pessi MA, Prochilo T, Quadri A, Gatta G et al (2010) Colon cancer. Critical Reviews in Oncology/Hematology 74(2):106– 133

    Article  Google Scholar 

  28. Li H, Chen D, Nailon WH, Davies ME, Laurenson D (2019) A deep dual-path network for improved mammogram image processing. In: ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 1224–1228

  29. Lin M, Chen Q, Yan S (2013) Network in network. arXiv:https://arxiv.org/abs/1312.4400

  30. Liu J, Li W, Zhao N, Cao K, Yin Y, Song Q, Chen H, Gong X (2018) Integrate domain knowledge in training cnn for ultrasonography breast cancer diagnosis. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 868–875

  31. Lundervold A, Lundervold A (2019) An overview of deep learning in medical imaging focusing on mri. Zeitschrift für Medizinische Physik 29(2):102–127

    Article  Google Scholar 

  32. Migliore L, Migheli F, Spisni R, Coppedè F. (2011) Genetics, cytogenetics, and epigenetics of colorectal cancer. BioMed Res Int, 2011

  33. Nair V, Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807–814

  34. Oh JE, Kim MJ, Lee J, Hur BY, Kim B, Kim DY, Baek JY, Chang HJ, Park SC, Oh JH et al (2019) Magnetic resonance-based texture analysis differentiating kras mutation status in rectal cancer. Cancer Research and Treatment

  35. Pal A, Balasubramanian VN (2019) Zero-shot task transfer. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2189–2198

  36. Schlemper J, Oktay O, Schaap M, Heinrich M, Kainz B, Glocker B, Rueckert D (2019) Attention gated networks: learning to leverage salient regions in medical images. Med Image Anal 53:197–207

    Article  Google Scholar 

  37. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626

  38. Shin HC, Tenenholtz NA, Rogers JK, Schwarz CG, Senjem ML, Gunter JL, Andriole KP, Michalski M (2018) Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 1–11

  39. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:https://arxiv.org/abs/1409.1556

  40. Sobel I, Feldman G (1968) A 3x3 isotropic gradient operator for image processing a talk at the Stanford Artificial Project in pp 271–272

  41. Song C, Huang Y, Ouyang W, Wang L (2018) Mask-guided contrastive attention model for person re-identification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1179–1188

  42. Sun Q, Liu Y, Chua TS, Schiele B (2019) Meta-transfer learning for few-shot learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 403–412

  43. Suzuki K (2017) Overview of deep learning in medical imaging. Radiol Phys Technol 10(3):257–273

    Article  Google Scholar 

  44. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  45. Torre LA, Bray F, Siegel RL, Ferlay J, Lortet-Tieulent J, Jemal A (2015) Global cancer statistics, 2012. CA: A Cancer Journal for Clinicians 65(2):87–108

    Google Scholar 

  46. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Kaiser Ł (2017) Attention is all you need. In: Advances in neural information processing systems , pp 5998–6008

  47. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3156–3164

  48. Wu X, Li Y, Chen X, Huang Y, He L, Ke Z, Huang X, Cheng Z, Zhang W, Huang Y et al (2019) Deep learning features improves the performance of hand-crafted radiomics signature for prediction of kras status in patients with colorectal cancer

  49. Xie Y, Xia Y, Zhang J, Song Y, Feng D, Fulham M, Cai W (2018) Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest ct. IEEE Trans Med Imag 38(4):991–1004

    Article  Google Scholar 

  50. Xu Y, Xu Q, Sun H, Liu T, Shi K, Wang W (2018) Could ivim and adc help in predicting the kras status in patients with rectal cancer? Europ Radiol 28(7):3059–3065

    Article  Google Scholar 

  51. Yang L, Dong D, Fang M, Zhu Y, Zang Y, Liu Z, Zhang H, Ying J, Zhao X, Tian J (2018) Can ct-based radiomics signature predict kras/nras/braf mutations in colorectal cancer? Europ Radiol 28 (5):2058–2067

    Article  Google Scholar 

  52. Zamir AR, Sax A, Shen W, Guibas LJ, Malik J, Savarese S (2018) Taskonomy: disentangling task transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3712–3722

  53. Zhang J, Xie Y, Wu Q, Xia Y (2019) Medical image classification using synergic deep learning. Med Image Anal 54:10–19

    Article  Google Scholar 

  54. Zhao A, Balakrishnan G, Durand F, Guttag JV, Dalca AV (2019) Data augmentation using learned transformations for one-shot medical image segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition , pp 8543–8553

Download references

Author information

Author notes

    Authors and Affiliations

    1. College of Information and Computer, Taiyuan University of Technology, Taiyuan, China

      JiaWen Wang, GuoHua Shi, JuanJuan Zhao, Yan Qiang, QianQian Du, Yue Ma & Ntikurako Guy-Fernand Kazihise

    2. Department of Radiology, Shanxi Province Cancer Hospital, Shanxi Medical University, Taiyuan, China

      YanFen Cui & XiaoTang Yang

    Authors
    1. JiaWen Wang
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. YanFen Cui
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. GuoHua Shi
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. JuanJuan Zhao
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. XiaoTang Yang
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Yan Qiang
      View author publications

      You can also search for this author in PubMed Google Scholar

    7. QianQian Du
      View author publications

      You can also search for this author in PubMed Google Scholar

    8. Yue Ma
      View author publications

      You can also search for this author in PubMed Google Scholar

    9. Ntikurako Guy-Fernand Kazihise
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding authors

    Correspondence to JuanJuan Zhao or XiaoTang Yang.

    Ethics declarations

    Conflict of interests

    The authors declare that they have no conflict of interest.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    JiaWen Wang and YanFen Cui contributed equally to this work.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Wang, J., Cui, Y., Shi, G. et al. Multi-branch cross attention model for prediction of KRAS mutation in rectal cancer with t2-weighted MRI. Appl Intell 50, 2352–2369 (2020). https://doi.org/10.1007/s10489-020-01658-8

    Download citation

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s10489-020-01658-8

    Keywords

    Search

    Navigation