Skip to main content
Springer Nature Link
Log in

Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The fusion of multimodal medical images has garnered painstaking attention for clinical diagnosis and surgical planning. Although various scholars have designed numerous fusion methods, the challenges of extracting substantial features without introducing noise and non-uniform contrast hindered the overall quality of fused photos. This paper presents a multimodal medical image fusion (MMIF) using a novel deep convolutional neural network (D-CNN) along with preprocessing schemes to circumvent the mentioned issues. A non-linear average median filtering (NL-AMF) and multiscale improved top-hat (MI-TH) approach are utilized at the preprocessing stage to remove noise and improve the contrast of images. The non-linear anisotropic diffusion (NL-AD) scheme is employed to split the photos into base and detailed parts. The fusion of base parts is accomplished by a dimension reduction method to retain the energy information. In contrast, the detailed parts are fused by novel D-CNN to preserve the enriched detailed features effectively. The simulation results demonstrate that the proposed method produces better brightness contrast and more image details than existing methods by acquiring 0.7649 to 0.8986, 0.3520 to 0.4783, 0.7639 to 0.9056, 68.8932 to 81.0487 gain for quality transfer ratio from source photo to a generated photo (\(Q_{G}^{AB}\)), feature mutual information (FMI), structural similarity index (SSIM), and average pixel intensity (API) respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Karim S, Tong G, Li J, Qadir A, Farooq U, Yiting Y (2023) Current advances and future perspectives of image fusion: a comprehensive review. Information Fusion 90:185–217. https://doi.org/10.1016/j.inffus.2022.09.019

    Article  Google Scholar 

  2. Aamir M, Rahman Z, Dayo ZA, Abro WA, Irfan Uddin M, Khan I, Imran AS, Ali Z, Ishfaq M, Guan Y, Zhihua H (2021) A deep learning approach for brain tumor classification using MRI images. Comput Electr Eng 101:108105. https://doi.org/10.1016/j.compeleceng.2022.108105

    Article  Google Scholar 

  3. Hermessi H, Mourali O, Zagrouba EJSP (2021) Multimodal medical image fusion review: theoretical background and recent advances. Signal Process 183:108036. https://doi.org/10.1016/j.sigpro.2021.108036

    Article  Google Scholar 

  4. Bhutto JA, Tian L, Qiliang D, Sun Z, Lubin Y, Soomro TA (2022) An improved infrared and visible image fusion using an adaptive contrast enhancement method and deep learning network with transfer learning. Remote Sens 14(4):939. https://doi.org/10.3390/rs14040939

    Article  Google Scholar 

  5. Azam MA, Khan KB, Salahuddin S, Rehman E, Khan SA, Khan MA, Kadry S, Gandomi AH (2022) A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics. Comput Biol Med 144:105253. https://doi.org/10.1016/j.compbiomed.2022.105253

    Article  Google Scholar 

  6. Almasri MM, Alajlan AM (2022) Artificial intelligence-based multimodal medical image fusion using hybrid S2 optimal CNN. Electronics 11(14):2124. https://doi.org/10.3390/electronics11142124

    Article  Google Scholar 

  7. Guan Y, Aamir M, Rahman Z, Ali A, Abro WA, Dayo ZA, Bhutta MS, Zhihua H (2021) A framework for efficient brain tumor classification using MRI images[J]. Math Biosci Eng 18(5):5790–5815. https://www.aimspress.com/article/doi/10.3934/mbe.2021292

    Article  Google Scholar 

  8. Li Y, Zhao J, Lv Z, Li J (2021) Medical image fusion method by deep learning. Int J Cogn Comput Eng 2:21–29. https://doi.org/10.1016/j.ijcce.2020.12.004

    Article  Google Scholar 

  9. Kong W, Li C, Lei Y (2022) Multimodal medical image fusion using convolutional neural network and extreme learning machine. Front Neurorobot 16:1050981. https://doi.org/10.3389/fnbot.2022.1050981

    Article  Google Scholar 

  10. Lou X-C, Feng XJC, Juhola M (2021) Medicine, multimodal medical image fusion based on multiple latent low-rank representation. Comput Math Methods Med:1–16. https://doi.org/10.1155/2021/1544955

  11. Wang X, Hua Z, Li J (2023) Multi-focus image fusion framework based on transformer and feedback mechanism. Ain Shams Eng J 14(5):101978. https://doi.org/10.1016/j.asej.2022.101978

    Article  Google Scholar 

  12. Zhou T, Li Q, Huiling L, Cheng Q, Zhang X (2023) GAN review: models and medical image fusion applications. Inform Fusion 91:134–148. https://doi.org/10.1016/j.inffus.2022.10.017

    Article  Google Scholar 

  13. Soomro TA, Khan TM, Khan MAU, Gao J, Paul M, Zheng L (2018) Impact of ICA-based image enhancement technique on retinal blood vessels segmentation. IEEE Access 6:3524–3538. https://doi.org/10.1109/ACCESS.2018.2794463

    Article  Google Scholar 

  14. Aamir M, Yi-Fei P, Rahman Z, Tahir M, Naeem H, Dai Q (2018) A framework for automatic building detection from low-contrast satellite images. Symmetry 11(1):3. https://doi.org/10.3390/sym11010003

    Article  Google Scholar 

  15. Bhutto JA, Lianfang T, Qiliang D, Soomro TA, Lubin Y, Tahir MF (2020) An enhanced image fusion algorithm by combined histogram equalization and fast gray level grouping using multi-scale decomposition and gray-PCA. IEEE Access 8:157005–157021. https://doi.org/10.1109/ACCESS.2020.3018264

    Article  Google Scholar 

  16. Ma W, Wang K, Li J, Yang SX, Li J, Song L, Li Q (2023) Infrared and visible image fusion technology and application: a review. Sensors 23(2):599. https://doi.org/10.3390/s23020599

    Article  Google Scholar 

  17. Choudhary G, Sethi D (2022) From conventional approach to machine learning and deep learning approach: an experimental and comprehensive review of image fusion techniques. Arch Comput Methods Eng 30:1267–1304 https://link.springer.com/article/10.1007/s11831-022-09833-5

    Article  Google Scholar 

  18. Tawade L, Aboobacker AB, Ghante F (2014) Image fusion based on wavelet transforms. Int J Bio-Sci Bio-Technol 6(3):149–162. https://doi.org/10.14257/ijbsbt.2014.6.3.18

    Article  Google Scholar 

  19. Shreyamsha Kumar BK (2013) (2012), multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform, in signal. Image Video Process 7:1125–1143 https://link.springer.com/article/10.1007/s11760-012-0361-x

    Article  Google Scholar 

  20. Gao G, Xu L, Dongzhu F (2013) Multi-focus image fusion based on non-subsampled shearlet transform. IET Image Process 7(6):543–639. https://doi.org/10.1049/iet-ipr.2012.0558

    Article  Google Scholar 

  21. Tawfik N, Elnemr HA, Fakhr M, Dessouky MI, Abd El-Samie FE (2022) Multimodal medical image fusion using stacked auto-encoder in NSCT domain. J Digit Imaging 35:1308–1325 https://link.springer.com/article/10.1007/s10278-021-00554-y

    Article  Google Scholar 

  22. Nagaraja Kumar N, Jayachandra Prasad T, Satya Prasad K (2022) An intelligent multimodal medical image fusion model based on improved fast discrete Curvelet transform and Type-2 fuzzy entropy. Int J Fuzzy Syst 25:96–117 https://link.springer.com/article/10.1007/s40815-022-01379-9

    Article  Google Scholar 

  23. Shreyamsha Kumar BK (2013) Image fusion based on pixel significance using cross bilateral filter. SIViP 9:1193–1204 https://link.springer.com/article/10.1007/s11760-013-0556-9

    Article  Google Scholar 

  24. Liu Y, Chen X, Ward RK, Jane Wang Z (2016) Image fusion with convolutional sparse representation. IEEE Signal Process Lett 23(12):1882–1886. https://doi.org/10.1109/LSP.2016.2618776

    Article  Google Scholar 

  25. Li X, Zhang X, Ding M (2019) A sum-modified-Laplacian and sparse representation based multimodal medical image fusion in Laplacian pyramid domain. Med Biol Eng Comput 57:2265–2275 https://link.springer.com/article/10.1007/s11517-019-02023-9

    Article  Google Scholar 

  26. Rajalingam B, Fadi Al-Turjman R, Santhoshkumar MR (2020) Intelligent multimodal medical image fusion with deep guided filtering. Multimed Syst 28:1449–1463. https://link.springer.com/article/10.1007/s00530-020-00706-0

    Article  Google Scholar 

  27. Wang L, Dou J, Qin P, Lin S, Gao Y, Wang R, Zhang J (2021) Multimodal medical image fusion based on nonsubsampled shearlet transform and convolutional sparse representation. Multimed Tools Appl 80:36401–36421. https://link.springer.com/article/10.1007/s11042-021-11379-w

    Article  Google Scholar 

  28. Guo P, Xie G, Li R, Hui H (2022) Multimodal medical image fusion with convolution sparse representation and mutual information correlation in NSST domain. Complex Intell Syst 9:317–328 https://link.springer.com/article/10.1007/s40747-022-00792-9

    Article  Google Scholar 

  29. Li H, Wu X-J (2018) Infrared and visible image fusion using latent low-rank representation. Comput Vis Pattern Recognit 5:6. https://doi.org/10.48550/arXiv.1804.08992

    Article  Google Scholar 

  30. Tawfik N, Elnemr HA, Fakhr M, Dessouky MI, Abd El-Samie FE (2021) Hybrid pixel-feature fusion system for multimodal medical images. J Ambient Intell Humaniz Comput 12:6001–6018. https://link.springer.com/article/10.1007/s12652-020-02154-0

    Article  Google Scholar 

  31. Venkatesan B, Ragupathy US (2022) Integrated fusion framework using hybrid domain and deep neural network for multimodal medical images. Multidim Syst Sign Process 33:819–834 https://link.springer.com/article/10.1007/s11045-021-00813-9

    Article  Google Scholar 

  32. Soomro TA, Afifi AJ, Gao J, Hellwich O, Zheng L, Paul M (2019) Strided fully convolutional neural network for boosting the sensitivity of retinal blood vessels segmentation. Expert Syst Appl 134(15):36–52. https://doi.org/10.1016/j.eswa.2019.05.029

    Article  Google Scholar 

  33. Aamir M, Rahman Z, Abro WA, Tahir M, Ahmed SM (2019) An optimized architecture of image classification using convolutional NeuralNetwork. Int J Image Graph Signal Process 10:30–39 https://www.mecs-press.org/ijigsp/ijigsp-v11-n10/IJIGSP-V11-N10-5.pdf

    Article  Google Scholar 

  34. Soomro TA, Afifi AJ, Zheng L, Soomro S, Gao J, Hellwich O, Paul M (2019) Deep learning models for retinal blood vessels segmentation: a review. IEEE Access 7:71696–71717. https://doi.org/10.1109/ACCESS.2019.2920616

    Article  Google Scholar 

  35. Zhang L, Li H, Zhu R, Ping D (2022) An infrared and visible image fusion algorithm based on ResNet-152. Multimed Tools Appl 81:9277–9287. https://link.springer.com/article/10.1007/s11042-021-11549-w

    Article  Google Scholar 

  36. Feng Y, Houqing L, Bai J, Cao L, Yin H (2020) Fully convolutional network-based infrared and visible image fusion. Multimed Tools Appl 79:15001–15014 https://link.springer.com/article/10.1007/s11042-019-08579-w

    Article  Google Scholar 

  37. Xu L, Jimmy SJ, Ren CL, Jia J (2014) Deep convolutional neural network for image deconvolution. In: Advances in neural information processing systems 27 (NIPS 2014). https://proceedings.neurips.cc/paper/2014/hash/1c1d4df596d01da60385f0bb17a4a9e0-Abstract.html

    Google Scholar 

  38. Krokos V, Xuan VB, Bordas SPA, Young P, Kerfriden P (2021) A Bayesian multiscale CNN framework to predict local stress fields in structures with microscale features. Comput Mech 69:733–766 https://link.springer.com/article/10.1007/s00466-021-02112-3

    Article  MathSciNet  Google Scholar 

  39. Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639. https://doi.org/10.1109/34.56205

    Article  Google Scholar 

  40. Bhutto JA, Tian L, Qiliang D, Sun Z, Lubin Y, Tahir MF (2022) CT and MRI medical image fusion using noise-removal and contrast enhancement scheme with convolutional neural network. Entropy 24(3):393. https://doi.org/10.3390/e24030393

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Huanggang Normal University, Huanggang, 438000, China

    Jameel Ahmed Bhutto, Jiang Guosong, Ziaur Rahman & Muhammad Ishfaq

  2. School of Automation Science and Engineering, South China University of Technology, Guangzhou, 510640, China

    Zhengzheng Sun

  3. Electronic Engineering Department, University of Larkano, Larkana, Sindh, Pakistan

    Toufique Ahmed Soomro

Authors
  1. Jameel Ahmed Bhutto
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jiang Guosong
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ziaur Rahman
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Muhammad Ishfaq
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Zhengzheng Sun
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Toufique Ahmed Soomro
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Jiang Guosong.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest in this research work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bhutto, J.A., Guosong, J., Rahman, Z. et al. Feature extraction of multimodal medical image fusion using novel deep learning and contrast enhancement method. Appl Intell 54, 5907–5930 (2024). https://doi.org/10.1007/s10489-024-05431-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05431-z

Keywords