Skip to main content

Advertisement

SpringerLink
Log in

A review on multimodal medical image fusion towards future research

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Image fusion is a technique used to merge two or more source images into a single image that incorporates more details than the originals and still offering an accurate depiction about the captured information. Resultant fused images are more accurate and provide comprehensive information for both the human and machine vision perception for further processing of the image. Image fusion provides better performance in the areas like pattern recognition, image processing, computer vision, machine learning and artificial intelligence. In the recent years image fusion has moved out of the laboratories and used in the real time applications. This paper provides the insight of various techniques for image fusion like primitive fusion (Simple averaging, Maxima and Minima, etc.), Discrete Wavelet Transform (DWT) based fusion, Principal Component Analysis (PCA) based fusion, Curvelet transform based fusion etc. On-going through various literatures, it is found that image fusion in spatial domain provides high resolution images, although the fusion algorithms are dependent on the nature of image and also depends on the application for which the image is to be fused. Hence, spectral domain fusion and hybrid fusion techniques are introduced and it is proven to be better than the spatial domain fusion. Comparison of all the techniques along with recent approaches are done to find the best approach towards future research to provide new direction to the researchers in medical sector.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Akbarpour T et al (2019) Medical image fusion based on nonsubsampled shearlet transform and principal component averaging. Int J Wavelets Multiresolut Inf Process 17(04):1950023

    Article  MATH  Google Scholar 

  2. Aktar MN, Lambert AJ, Pickering M (2018) An automatic fusion algorithm for multi-modal medical images. Comput Methods Biomech Biomed Eng: Imaging Vis 6(5):584–598

    Google Scholar 

  3. Algarni AD (2020) Automated medical diagnosis system based on multi-modality image fusion and deep learning. Wirel Pers Commun 111(2):1033–1058

    Article  Google Scholar 

  4. Amin-Naji M, Aghagolzadeh A, Ezoji M (2019) Ensemble of CNN for multi-focus image fusion. Inf fusion 51:201–214

    Article  Google Scholar 

  5. Anandhi D, Valli S (2018) An algorithm for multi-sensor image fusion using maximum a posteriori and nonsubsampled contourlet transform. Comput Electr Eng 65:139–152

    Article  Google Scholar 

  6. Arathy Menon NP, Arunvinodh C, Davis AM (2015) Comparative analysis of transform based image fusion techniques for medical applications. In: 2015 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS). IEEE

  7. Anilkumar B, Kumar PR (2020) Multi tumor classification in MR brain images through deep feature extraction using CNN and supervised classifier. Int J Emerg Technol 11(1):83–90

  8. Arif M, Wang G (2020) Fast curvelet transform through genetic algorithm for multimodal medical image fusion. Soft Comput 24(3):1815–1836

    Article  Google Scholar 

  9. Auer T et al (2019) Fusion imaging of contrast-enhanced ultrasound with CT or MRI for kidney lesions. In Vivo 33:203–2081

    Article  Google Scholar 

  10. Aymaz S, Köse C (2019) A novel image decomposition-based hybrid technique with super-resolution method for multi-focus image fusion. Inf Fusion 45:113–127

    Article  Google Scholar 

  11. Aymaz S, Köse C, Aymaz Ş (2020) Multi-focus image fusion for different datasets with super-resolution using gradient-based new fusion rule. Multimed Tools Appl 79(19):13311–13350

    Article  Google Scholar 

  12. Barba-J L et al (2019) A hermite-based method for bone SPECT/CT image fusion with prior segmentation. In: ECCOMAS thematic conference on computational vision and medical image processing. Springer

  13. Benjamin JR, Jayasree T (2018) Improved medical image fusion based on cascaded PCA and shift invariant wavelet transforms. Int J Comput Assist Radiol Surg 13(2):229–240

    Article  Google Scholar 

  14. Bhatnagar G, Wu QJ, Liu Z (2013) Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans Multimedia 15(5):1014–1024

    Article  Google Scholar 

  15. Bhatnagar G, Wu QJ, Liu Z (2015) A new contrast based multimodal medical image fusion framework. Neurocomputing 157:143–152

    Article  Google Scholar 

  16. Bhavana V, Krishnappa H (2015) Multi-modality medical image fusion using discrete wavelet transform. Procedia Comput Sci 70:625–631

    Article  Google Scholar 

  17. Chavan S, Pawar A, Talbar S (2017) Multimodality medical image fusion using rotated wavelet transform. Adv Intell Syst Res 137:627–635

    Google Scholar 

  18. Chavan SS et al (2017) Nonsubsampled rotated complex wavelet transform (NSRCxWT) for medical image fusion related to clinical aspects in neurocysticercosis. Comput Biol Med 81:64–78

    Article  Google Scholar 

  19. Daneshvar S, Ghassemian H (2010) MRI and PET image fusion by combining IHS and retina-inspired models. Inf Fusion 11(2):114–123

    Article  Google Scholar 

  20. Ding Z et al (2021) Siamese networks and multi-scale local extrema scheme for multimodal brain medical image fusion. Biomed Signal Process Control 68:102697

    Article  Google Scholar 

  21. Dogra A et al (2017) Efficient fusion of osseous and vascular details in wavelet domain. Pattern Recognit Lett 94:189–193

    Article  Google Scholar 

  22. Du C, Gao S (2018) Multi-focus image fusion algorithm based on pulse coupled neural networks and modified decision map. Optik 157:1003–1015

    Article  Google Scholar 

  23. Du J, Li W (2020) Two-scale image decomposition based image fusion using structure tensor. Int J Imaging Syst Technol 30(2):271–284

    Article  Google Scholar 

  24. El-Hoseny HM et al (2018) An efficient DT-CWT medical image fusion system based on modified central force optimization and histogram matching. Infrared Phys Technol 94:223–231

    Article  Google Scholar 

  25. El-Hoseny HM et al (2019) An optimal wavelet-based multi-modality medical image fusion approach based on modified central force optimization and histogram matching. Multimed Tools Appl 78(18):26373–26397

    Article  Google Scholar 

  26. Fu J et al (2021) Multimodal biomedical image fusion method via rolling guidance filter and deep convolutional neural networks. Optik 237:166726

    Article  Google Scholar 

  27. Ganasala P, Kumar V (2016) Feature-motivated simplified adaptive PCNN-based medical image fusion algorithm in NSST domain. J Digit Imaging 29(1):73–85

    Article  Google Scholar 

  28. Gomathi PS, Kalaavathi B (2016) Multimodal medical image fusion in non-subsampled contourlet transform domain. Circuits Syst 7(08):1598

    Article  Google Scholar 

  29. Gupta D (2018) Nonsubsampled shearlet domain fusion techniques for CT–MR neurological images using improved biological inspired neural model. Biocybern Biomed Eng 38(2):262–274

    Article  Google Scholar 

  30. Haghighat MBA, Aghagolzadeh A, Seyedarabi H (2011) A non-reference image fusion metric based on mutual information of image features. Comput Electr Eng 37(5):744–756

    Article  MATH  Google Scholar 

  31. Hermessi H, Mourali O, Zagrouba E (2018) Convolutional neural network-based multimodal image fusion via similarity learning in the shearlet domain. Neural Comput Appl 30(7):2029–2045

    Article  Google Scholar 

  32. Hou R et al (2019) Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model. Med Biol Eng Comput 57(4):887–900

    Article  Google Scholar 

  33. Huang H, Feng X, Jiang J (2017) Medical image fusion algorithm based on nonlinear approximation of contourlet transform and regional features. J Electr Comput Eng 2017:6807473

    Google Scholar 

  34. Huang C et al (2019) A new pulse coupled neural network (PCNN) for brain medical image fusion empowered by shuffled frog leaping algorithm. Front NeuroSci 13:210

    Article  Google Scholar 

  35. Huang B et al (2020) A review of multimodal medical image fusion techniques. Comput Math Methods Med 2020:8279342

    Article  Google Scholar 

  36. Jagalingam P, Hegde AV (2015) A review of quality metrics for fused image. Aquat Procedia 4:133–142

    Article  Google Scholar 

  37. Jin X et al (2016) Mixed criticality scheduling for industrial wireless sensor networks. Sensors 16(9):1376

    Article  Google Scholar 

  38. Jin X et al (2018) Multimodal sensor medical image fusion based on nonsubsampled shearlet transform and S-PCNNs in HSV space. Sig Process 153:379–395

    Article  Google Scholar 

  39. Kalita DJ, Singh VP, Kumar V (2022) Two-way threshold-based intelligent water drops feature selection algorithm for accurate detection of breast cancer. Soft Comput 26(5):2277–2305

    Article  Google Scholar 

  40. Kaur M, Singh D (2020) Fusion of medical images using deep belief networks. Cluster Comput 23(2):1439–1453

    Article  Google Scholar 

  41. Li C, Zhu A (2020) Application of image fusion in diagnosis and treatment of liver cancer. Appl Sci 10(3):1171

    Article  Google Scholar 

  42. Li W, Wang K, Cai K (2019) Medical image fusion based on saliency and adaptive similarity judgment. Pers Ubiquitous Comput: 1–7

  43. Li Y et al (2021) Medical image fusion method by deep learning. Int J Cogn Comput Eng 2:21–29

    Google Scholar 

  44. Liu Y et al (2017) A medical image fusion method based on convolutional neural networks. In: 2017 20th international conference on information fusion (fusion). IEEE

  45. Liu Y et al (2017) Multi-focus image fusion with a deep convolutional neural network. Inform Fusion 36:191–207

    Article  Google Scholar 

  46. Liu X, Mei W, Du H (2017) Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion. Neurocomputing 235:131–139

    Article  Google Scholar 

  47. Ma J et al (2016) Infrared and visible image fusion via gradient transfer and total variation minimization. Inf Fusion 31:100–109

    Article  Google Scholar 

  48. Manchanda M, Sharma R (2018) An improved multimodal medical image fusion algorithm based on fuzzy transform. J Vis Commun Image Represent 51:76–94

    Article  Google Scholar 

  49. Meng L, Guo X, Li H (2019) MRI/CT fusion based on latent low rank representation and gradient transfer. Biomed Signal Process Control 53:101536

    Article  Google Scholar 

  50. Miao Y, Chunyu N, Yazhuo X (2021) Brain medical image fusion scheme based on shuffled frog-leaping algorithm and adaptive pulse‐coupled neural network. IET Image Processing

  51. Naidu V (2010) Discrete cosine transform-based image fusion. Def Sci J 60(1):48

    Article  Google Scholar 

  52. Naveenadevi R, Nirmala S, Babu GT (2017) Fusion of CT-PET lungs tumour images using dual tree complex wavelet transform. Res J Pharm Biol Chem Sci 8(1):310–316

    Google Scholar 

  53. Nikolakopoulos K, Oikonomidis D (2015) Quality assessment of ten fusion techniques applied on worldview-2. Eur J Remote Sens 48(1):141–167

  54. Özyurt F et al (2019) Brain tumor detection based on convolutional neural network with neutrosophic expert maximum fuzzy sure entropy. Measurement 147:106830

  55. Parvathy VS, Pothiraj S (2020) Multi-modality medical image fusion using hybridization of binary crow search optimization. Health Care Manag Sci 23(4):661–669

    Article  Google Scholar 

  56. Patil U, Mudengudi U (2011) Image fusion using hierarchical PCA. In: 2011 international conference on image information processing. IEEE

  57. Patil HV, Shirbahadurkar SD (2018) FWFusion: fuzzy whale fusion model for MRI multimodal image fusion. Sādhanā 43(3):1–16

  58. Polinati S, Dhuli R (2019) A review on multi-model medical image fusion. In: 2019 International Conference on Communication and Signal Processing (ICCSP). IEEE

  59. Prakash C, Rajkumar S, Mouli PC (2012) Medical image fusion based on redundancy DWT and Mamdani type min-sum mean-of-max techniques with quantitative analysis. In: 2012 international conference on recent advances in computing and software systems and software systems. IEEE

  60. Prakash O et al (2019) Multiscale fusion of multimodal medical images using lifting scheme based biorthogonal wavelet transform. Optik 182:995–1014

    Article  Google Scholar 

  61. Rajalingam B, Priya R (2018) Multimodal medical image fusion based on deep learning neural network for clinical treatment analysis. Int J ChemTech Res 11(06):160–176

    Google Scholar 

  62. Rani K, Sharma R (2013) Study of different image fusion algorithm. Int J Emerg Technol Adv Eng 3(5):288–291

    Google Scholar 

  63. Ravi P, Krishnan J (2018) Image enhancement with medical image fusion using multiresolution discrete cosine transform. Mater Today: Proc 5(1):1936–1942

    Google Scholar 

  64. Sandhya S, Kumar MS, Karthikeyan L (2019) A hybrid fusion of multimodal medical images for the enhancement of visual quality in medical diagnosis. Computer aided intervention and diagnostics in clinical and medical images. Springer, pp 61–70

  65. Shabu SJ, Jayakumar DC, Surya T (2013) Survey of image fusion techniques for brain tumor detection. Int J Eng Adv Technol 3(2):457–459

    Google Scholar 

  66. Shahdoosti HR, Mehrabi A (2018) MRI and PET image fusion using structure tensor and dual ripplet-II transform. Multimed Tools Appl 77(17):22649–22670

    Article  Google Scholar 

  67. Shariaty F et al (2022) Texture appearance model, a new model-based segmentation paradigm, application on the segmentation of lung nodule in the CT scan of the chest. Comput Biol Med 140:105086

    Article  Google Scholar 

  68. Singh R, Khare A (2013) Multiscale medical image fusion in wavelet domain. Sci World J 2013:521034

    Article  Google Scholar 

  69. Singh S, Rajput R (2014) A comparative study of classification of image fusion techniques. Int J Eng Comput Sci 3:7350–7353

    Google Scholar 

  70. Singh RR, Mishra R (2015) Benefits of dual tree complex wavelet transform over discrete wavelet transform for image fusion. Int J Innovative Res Sci Technol 1(11):259–263

    Google Scholar 

  71. Song Z, Jiang H, Li S (2017) A novel fusion framework based on adaptive PCNN in NSCT domain for whole-body PET and CT images. Comput Math Methods Med 2017:8407019

    Article  MATH  Google Scholar 

  72. Sui Y et al (2019) Application value of MRI diffuse weighted imaging combined with PET/CT in the diagnosis of stomach cancer at different stages. Oncol Lett 18(1):43–48

    Google Scholar 

  73. Tan L, Yu X (2019) Medical image fusion based on fast finite shearlet transform and sparse representation. Comput Math Methods Med 2019:3503267

    Article  MATH  Google Scholar 

  74. Tang L et al (2017) Multimodal medical image fusion based on discrete T chebichef moments and pulse coupled neural network. Int J Imaging Syst Technol 27(1):57–65

    Article  Google Scholar 

  75. Tang H et al (2018) Pixel convolutional neural network for multi-focus image fusion. Inf Sci 433:125–141

    Article  Google Scholar 

  76. Tannaz A et al (2020) Fusion of multimodal medical images using nonsubsampled shearlet transform and particle swarm optimization. Multidimens Syst Signal Process 31(1):269–287

    Article  MATH  Google Scholar 

  77. Udomhunsakul S et al (2011) Multiresolution edge fusion using SWT and SFM. In: Proceedings of the world congress on engineering

  78. Verma A, Singh VP (2022) HSADML: hyper-sphere angular deep metric based learning for brain tumor classification. arXiv preprint arXiv:2201.12269

  79. Vijayarajan R, Muttan S (2015) Discrete wavelet transform based principal component averaging fusion for medical images. AEU-Int J Electron Commun 69(6):896–902

    Article  Google Scholar 

  80. Wang L et al (2019) An improved coupled dictionary and multi-norm constraint fusion method for CT/MR medical images. Multimed Tools Appl 78(1):929–945

    Article  Google Scholar 

  81. Wang Z et al (2019) Multifocus image fusion using convolutional neural networks in the discrete wavelet transform domain. Multimed Tools Appl 78(24):34483–34512

    Article  Google Scholar 

  82. Xia K, Yin, Wang J-q (2019) A novel improved deep convolutional neural network model for medical image fusion. Cluster Comput 22(1):1515–1527

    Article  Google Scholar 

  83. Xu X, Wang Y, Chen S (2016) Medical image fusion using discrete fractional wavelet transform. Biomed Signal Process Control 27:103–111

    Article  Google Scholar 

  84. Yakhdani MF, Azizi A (2010) Quality assessment of image fusion techniques for multisensor high resolution satellite images (case study: IRS-P5 and IRS-P6 satellite images). na

  85. Yang Y et al (2016) Multimodal sensor medical image fusion based on type-2 fuzzy logic in NSCT domain. IEEE Sens J 16(10):3735–3745

    Article  Google Scholar 

  86. Yin M et al (2017) A novel infrared and visible image fusion algorithm based on shift-invariant dual-tree complex shearlet transform and sparse representation. Neurocomputing 226:182–191

    Article  Google Scholar 

  87. Zhu Z et al (2018) A novel multi-modality image fusion method based on image decomposition and sparse representation. Inf Sci 432:516–529

    Article  Google Scholar 

  88. Zhou H (2012) An stationary wavelet transform and curvelet transform based infrared and visible images fusion algorithm. Int J Digit Content Technol Appl 6(1)

  89. Zuo Y et al (2017) Airborne infrared and visible image fusion combined with region segmentation. Sensors 17(5):1127

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electronics and Instrumentation Engineering, Kongu Engineering College, Erode, Tamilnadu, India

    B. Venkatesan & U. S. Ragupathy

  2. Department of Electrical and Electronics Engineering, Sri Shakthi Institute of Engineering and Technology, Coimbatore, Tamilnadu, India

    Indhu Natarajan

Authors
  1. B. Venkatesan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. U. S. Ragupathy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Indhu Natarajan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to B. Venkatesan.

Ethics declarations

Conflict of interest

 The authors certify that there is NO conflict of interest in relation to this work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Venkatesan, B., Ragupathy, U.S. & Natarajan, I. A review on multimodal medical image fusion towards future research. Multimed Tools Appl 82, 7361–7382 (2023). https://doi.org/10.1007/s11042-022-13691-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13691-5

Keywords

Search

Navigation