Skip to main content

Advertisement

Springer Nature Link
Log in

WAE-TLDN: self-supervised fusion for multimodal medical images via a weighted autoencoder and a tensor low-rank decomposition network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Multimodal medical image fusion (MMIF) integrates the advantages of multiple source images to assist clinical diagnosis. Existing image fusion methods need help to distinguish the importance between features and often define features to be retained subjectively, which leads to global structure loss and limits the performance of fusion. To overcome these restrictions, we propose a novel self-supervised tensor low-rank decomposition fusion network that can effectively extract global information from high-rank to low-rank conversion processes. Specifically, the compensation of textural features is performed by employing a self-supervised auxiliary task, and the whole network is dynamically fine-tuned according to a hybrid loss. In our model, an enhanced weights (EW) estimation method based on the global luminance contrast is developed, and a structure tensor loss with constraints is introduced to improve the robustness of the fusion results. Moreover, extensive experiments on six types of multimodal medical images show that visual and qualitative results are superior to competitors, validating the effectiveness of our methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. http://www.med.harvard.edu/AANLIB/home.html

References

  1. Ding AS, Lu A, Li Z, Galaiya D, Siewerdsen JH, Taylor RH, Creighton FX (2022) Automated registration-based temporal bone computed tomography segmentation for applications in neurotologic surgery. Otolaryngol Head Neck Surg 167(1):133–140

    Article  PubMed  Google Scholar 

  2. Wu HT, Zheng K, Huang Q, Hu J (2021) Contrast enhancement of multiple tissues in mr brain images with reversibility. IEEE Signal Process Lett 28:160–164. https://doi.org/10.1109/LSP.2020.3048840

    Article  ADS  Google Scholar 

  3. Valladares A, Beyer T, Rausch I (2020) Physical imaging phantoms for simulation of tumor heterogeneity in pet, ct, and mri: an overview of existing designs. Med Phys 47(4):2023–2037

    Article  PubMed  Google Scholar 

  4. Zhang M, Chu C, Huang L, Hu B (2022) Ct-mr image fusion for post-implant dosimetry analysis in brain tumor seed implantation-a preliminary study. Dis Markers 2022

  5. Liu X, Li W, Liu Z, Du F, Zou Q (2021) A dual-branch model for diagnosis of parkin-son’s disease based on the independent and joint features of the left and right gait. Appl Intell 1–12

  6. Říha P, Doležalová I, Mareček R, Lamoš M, Bartoňová M, Kojan M, Mikl M, Gajdoš M, Vojtíšek L, Bartoň M et al (2022) Multimodal combination of neuroimaging methods for localizing the epileptogenic zone in mr-negative epilepsy. Sci Rep 12(1):15158

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  7. Lu F, Du L, Chen W, Jiang H, Yang C, Pu Y, Wu J, Zhu J, Chen T, Zhang X et al (2022) T 1-t 2 dual-modal magnetic resonance contrast-enhanced imaging for rat liver fibrosis stage. RSC Adv 12(55):35809–35819

    Article  CAS  PubMed  PubMed Central  ADS  Google Scholar 

  8. Ullah H, Zhao Y, Abdalla FY, Wu L (2022) Fast local laplacian filtering based enhanced medical image fusion using parameter-adaptive pcnn and local features-based fuzzy weighted matrices. Appl Intell 1–20

  9. Yin M, Liu X, Liu Y, Chen X (2018) Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans Instrum Meas 68(1):49–64

    Article  ADS  Google Scholar 

  10. Vanitha K, Satyanarayana D, Prasad MG (2021) Multi-modal medical image fusion algorithm based on spatial frequency motivated pa-pcnn in the nsst domain. Curr Med Imaging Rev 17(5):634–643

    Article  CAS  Google Scholar 

  11. Nie R, Cao J, Zhou D, Qian W (2020) Multi-source information exchange encoding with pcnn for medical image fusion. IEEE Trans Circuits Syst Video Technol 31(3):986–1000

    Article  Google Scholar 

  12. Chen L, Wang X, Zhu Y, Nie R (2022) Multi-level difference information replenishment for medical image fusion. Appl Intell 1–13

  13. Zhao Z, Bai H, Zhang J, Zhang Y, Xu S, Lin Z, Timofte R, Van Gool L (2023) Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5906–5916

  14. Ding Z, Li H, Guo Y, Zhou D, Liu Y, Xie S (2023) M4fnet: Multimodal medical image fusion network via multi-receptive-field and multi-scale feature integration. Comput Biol Med 159:106923

    Article  PubMed  Google Scholar 

  15. Zhang G, Nie R, Cao J (2022) Ssl-waeie: Self-supervised learning with weighted auto-encoding and information exchange for infrared and visible image fusion. EEE/CAA J Autom Sin 9(9):1694–1697

    Article  Google Scholar 

  16. Zhang Y, Nie R, Cao J, Ma C, Wang C (2023) Ss-ssan: a self-supervised subspace attentional network for multi-modal medical image fusion. Artif Intell Rev 1–23

  17. Liang P, Jiang J, Liu X, Ma J (2022) Fusion from decomposition: A self-supervised decomposition approach for image fusion. In: European Conference on Computer Vision, pp. 719–735 Springer

  18. Aghamaleki JA, Ghorbani A (2023) Image fusion using dual tree discrete wavelet transform and weights optimization. Vis Comput 39(3):1181–1191

    Article  Google Scholar 

  19. Babu BS, Narayana MV (2023) Two stage multi-modal medical image fusion with marine predator algorithm-based cascaded optimal dtcwt and nsst with deep learning. Biomed Signal Process Control 85:104921

    Article  Google Scholar 

  20. Tan W, Tiwari P, Pandey HM, Moreira C, Jaiswal AK (2020) Multimodal medical image fusion algorithm in the era of big data. Neural Comput Appl 1–21

  21. Li X, Zhou F, Tan H (2021) Joint image fusion and denoising via three-layer decomposition and sparse representation. Knowl Based Syst 224:107087

    Article  Google Scholar 

  22. Li H, Xu T, Wu XJ, Lu J, Kittler J (2023) Lrrnet: A novel representation learning guided fusion network for infrared and visible images. IEEE Trans Pattern Anal Mach Intell

  23. Meng L, Guo X, Li H (2019) Mri/ct fusion based on latent low rank representation and gradient transfer. Biomed Signal Process Control 53:101536

    Article  Google Scholar 

  24. Li W, Peng X, Fu J, Wang G, Huang Y, Chao F (2022) A multiscale double-branch residual attention network for anatomical-functional medical image fusion. Comput Biol Med 141:105005

    Article  PubMed  Google Scholar 

  25. Xu H, Ma J, Jiang J, Guo X, Ling H (2020) U2fusion: A unified unsupervised image fusion network. IEEE Trans. Pattern Anal Mach Intell 44(1):502–518

    Article  Google Scholar 

  26. Zhang H, Ma J (2021) Sdnet: A versatile squeeze-and-decomposition network for real-time image fusion. Int J Comput Vis 129(10):2761–2785

    Article  Google Scholar 

  27. Zhang H, Xu H, Xiao Y, Guo X, Ma J (2020) Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity. Proceedings of the AAAI Conference on Artificial Intelligence 34:12797–12804

    Article  Google Scholar 

  28. Ma J, Xu H, Jiang J, Mei X, Zhang XP (2020) Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans Image Process 29:4980– 4995

  29. Fu J, Li W, Du J, Xu L (2021) Dsagan: A generative adversarial network based on dual-stream attention mechanism for anatomical and functional image fusion. Inf Sci 576(9)

  30. Huang J, Le Z, Ma Y, Fan F, Zhang H, Yang L (2020) Mgmdcgan: Medical image fusion using multi-generator multi-discriminator conditional generative adversarial network. IEEE Access 8:55145–55157

    Article  Google Scholar 

  31. Li X, Guo X, Han P, Wang X, Li H, Luo T (2020) Laplacian redecomposition for multimodal medical image fusion. IEEE Trans Instrum Meas 69(9):6880–6890

    Article  ADS  Google Scholar 

  32. Liu X, Zhang B, Li X, Liu S, Yue C, Liang SY (2023) An approach for tool wear prediction using customized densenet and gru integrated model based on multi-sensor feature fusion. J Intell Manuf 34(2):885–902

    Article  Google Scholar 

  33. Zhang G, Nie R, Cao J, Chen L, Zhu Y (2023) Fdgnet: A pair feature difference guided network for multimodal medical image fusion. Biomed Signal Process Control 81:104545

    Article  Google Scholar 

  34. Zhang B, Wang Y, Ding C, Deng Z, Li L, Qin Z, Ding Z, Bian L, Yang C (2023) Multi-scale feature pyramid fusion network for medical image segmentation. Int J Comput Assist Radiol Surg 18(2):353–365

    Article  PubMed  Google Scholar 

  35. Liu L, Zhou Y, Huo J, Wu Y, Gu R (2023) Heterogenous image fusion model with sr-dual-channel pcnn significance region for nsst in an apple orchard. Appl Intell 1–22

  36. Goyal S, Singh V, Rani A, Yadav N (2020) Fprsgf denoised non-subsampled shearlet transform-based image fusion using sparse representation. Signal Image Video Process 14:719–726

    Article  Google Scholar 

  37. Dinh PH (2021) Multi-modal medical image fusion based on equilibrium optimizer algorithm and local energy functions. Appl Intell 51(11):8416–8431

    Article  Google Scholar 

  38. Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM review 51(3):455–500

    Google Scholar 

  39. Fu Z, Zhao Y, Chang D, Wang Y, Wen J (2022) Latent low-rank representation with weighted distance penalty for clustering. IEEE Trans Cybern 1–13. https://doi.org/10.1109/TCYB.2022.3166545

  40. Zhao X, Yu Y, Zhou G, Zhao Q, Sun W (2022) Fast hypergraph regularized nonnegative tensor ring decomposition based on low-rank approximation. Appl Intell 1–24

  41. Wang B, Niu H, Zeng J, Bai G, Lin S, Wang Y (2021) Latent representation learning model for multi-band images fusion via low-rank and sparse embedding. IEEE Trans Multimedia 23:3137–3152. https://doi.org/10.1109/TMM.2020.3020695

    Article  Google Scholar 

  42. Woo S, Park J, Lee JY, Kweon IS (2018) Cbam: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19

  43. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.7132–7141

  44. Song Q, Li J, Li C, Guo H, Huang R (2022) Fully attentional network for semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence 36:2280–2288

    Article  Google Scholar 

  45. Chen W, Zhu X, Sun R, He J, Li R, Shen X, Yu B (2020) Tensor low-rank reconstruction for semantic segmentation. In: European Conference on Computer Vision, pp. 52–69 Springer

  46. Senhaji K, Ramchoun H, Ettaouil M (2020) Training feedforward neural network via multiobjective optimization model using non-smooth l1/2 regularization. Neurocomputing 410:1–11

    Article  Google Scholar 

  47. Mo Y, Wu Y, Yang X, Liu F, Liao Y (2022) Review the state-of-the-art technologies of semantic segmentation based on deep learning. Neurocomputing 493:626–646

    Article  Google Scholar 

  48. Cheng S, Wang Y, Huang H, Liu D, Liu S (2020) Nbnet: Noise basis learning for image denoising with subspace projection

  49. Inanici MN, Navvab M (2006) The virtual lighting laboratory: Per-pixel luminance data analysis. Leukos 3(2):89–104

    Article  Google Scholar 

  50. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4):600–612

    Article  PubMed  ADS  Google Scholar 

  51. Jung H, Kim Y, Jang H, Ha N, Sohn K (2020) Unsupervised deep image fusion with structure tensor representations. IEEE Trans Image Process 29:3845–3858

    Article  ADS  Google Scholar 

  52. Bhandari M, Parajuli P, Chapagain P, Gaur L (2021) Evaluating performance of adam optimization by proposing energy index. In: International Conference on Recent Trends in Image Processing and Pattern Recognition, pp. 156–168 Springer

  53. Piella G, Heijmans H (2003) A new quality metric for image fusion. In: Proceedings 2003 International Conference on Image Processing (Cat. No. 03CH37429), vol. 3, p. 173 IEEE

  54. Han Y, Cai Y, Cao Y, Xu X (2013) A new image fusion performance metric based on visual information fidelity. Inf Fusion 14(2):127–135

    Article  Google Scholar 

  55. Zhang L, Zhang L, Mou X, Zhang D (2011) Fsim: A feature similarity index for image quality assessment. IEEE Trans Image Process 20(8):2378–2386

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  56. Hossny M, Nahavandi S, Creighton D (2008) Comments on’information measure for performance of image fusion. Electron Lett 44(18):1066–1067

    Article  ADS  Google Scholar 

  57. Qu G, Zhang D, Yan P (2002) Information measure for performance of image fusion. Electron Lett 38(7):1

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants 61966037, 61833005, and 61463052, the China Postdoctoral Science Foundation under Grant 2017M621586, the Program of Yunnan Key Laboratory of Intelligent Systems and Computing (202205AG070003), and the Postgraduate Science Foundation of Yunnan University under Grants 2021Y263 and ZC-22222078. Key Project of Yunnan Basic Research Program under grant 202301AS070025 and Project Fund of Yunnan Provincial Department of Science and Technology 202105AF150011.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yunnan University, East Outer Ring South Road, Kunming, Yunnan, 650500, China

    Linna Pan, Rencan Nie, Gucheng Zhang & Yao Han

  2. School of Mathematics, Southeast University, Four archways, Nanjing, Jiangsu, 210096, China

    Rencan Nie & Jinde Cao

  3. Yunan Key Laboratory of Intelligent Systems and Computing, East Outer Ring South Road, Kunming, Yunnan, 650500, China

    Rencan Nie

  4. Yonsei Frontier Laboratory, Yonsei University, Yonsei-ro, Seoul, Seodaemun-gu, 03722, South Korea

    Jinde Cao

Authors
  1. Linna Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rencan Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gucheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinde Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yao Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rencan Nie.

Ethics declarations

Competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pan, L., Nie, R., Zhang, G. et al. WAE-TLDN: self-supervised fusion for multimodal medical images via a weighted autoencoder and a tensor low-rank decomposition network. Appl Intell 54, 1656–1671 (2024). https://doi.org/10.1007/s10489-023-05097-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05097-z

Keywords