Skip to main content

Advertisement

MSDCNet: Multi-stage and deep residual complementary multi-focus image fusion network based on multi-scale feature learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Addressing the boundary blurring problem in focus and out-of-focus regions is a key area of research in multifocus image fusion. Effective utilization of multi-scale modules is essential for enhancing performance. Therefore, this paper proposes a multi-stage feature extraction and deep residual complementary multifocus image fusion network. In the feature extraction stage, the V-shaped connection module captures the main objects and contours of the image. The feature thinning extraction module uses extended convolution to learn image details and refine textures at multiple scales. The advanced feature texture enhancement module targets boundary blurring regions, enhancing texture details and improving fusion quality. Asymmetric convolution reduces the network’s computational burden, improving feature learning efficiency. The fusion strategy uses a compound loss function to ensure image quality and prevent color distortion. The image reconstruction module uses residual connections with different-sized convolution kernels to maintain feature consistency and improve image quality. The network utilizes a dual-path Pseudo-Siamese structure, which handles image focus and defocus regions separately. Experimental results demonstrate the algorithm’s effectiveness. On the Lytro dataset, it achieves AG and EI metric values of 6.9 and 72.5, respectively, outperforming other methods. Fusion metrics SD = 61.80, SF = 19.63, and VIF = 0.94 surpass existing algorithms, effectively resolving the boundary blurring problem and providing better visual perception and broader applicability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets generated during and analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Li H, Wu XJ, Kittler J (2020) MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Trans Image Process 29:4733–4746

    Article  MATH  Google Scholar 

  2. Yang P, Gao LF, Zi LL (2021) “Image fusion method of convolution sparsity and detail saliency map analysis. Journal of Image and Graphics 26(10):2433–2449

    Article  MATH  Google Scholar 

  3. Huo X, Zou Y, Chen Y, Tan JQ (2021) Dual-scale decomposition and saliency analysis based infrared and visible image fusion. Journal of Image and Graphics 26(12):2813–2825

    Article  MATH  Google Scholar 

  4. Li S, Kwok JT, Wang Y (2001) ‘Combination of images with diverse focuses using the spatial frequency.’ Inf Fusion 2(3):169–176

    Article  MATH  Google Scholar 

  5. Li S, Kwok JT, Wang Y (2002) ‘Multifocus image fusion using artifificial neural networks.’ Pattern Recognit Lett 23(8):985–997

    Article  MATH  Google Scholar 

  6. Geng T, Liu XY, Wang X et al (2021) Deep shearlet residual learning network for single image super-resolution. IEEE Trans Image Process 30:4129–4142

    Article  MathSciNet  MATH  Google Scholar 

  7. Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense SIFT’. Inf Fusion 23:139–155

    Article  MATH  Google Scholar 

  8. Tang H, Xiao B, Li W, Wang G (2018) Pixel convolutional neural network for multi-focus image fusion’. Inf Sci 433–434:125–141

    Article  MathSciNet  MATH  Google Scholar 

  9. Amin-Naji M, Aghagolzadeh A, Ezoji M (2019) Ensemble of CNN for multi-focus image fusion’. Inf Fusion 51:201–214

    Article  MATH  Google Scholar 

  10. Amin-Naji M, Aghagolzadeh A, Ezoji M (2020) CNNs hard voting for multi-focus image fusion[J]. J Ambient Intell Humanized Comput 11(4):1749–1769

    Article  Google Scholar 

  11. Ma JY, Tang LF, Xu ML, Zhang H, Xiao GB (2021) STDFusionNet: an infrared and visible image fusion network based on salient target detection. IEEE Trans Instrum Meas 70:1–13

    MATH  Google Scholar 

  12. Wang YC, Xu S, Liu JM, Zhao ZX, Zhang CX, Zhang JS (2021) MFIF-GAN: a new generative adversarial network for multifocus image fusion. Signal Processing: Image Communication 96:116295

    MATH  Google Scholar 

  13. Zhang H, Le ZL, Shao ZF, Xu H, Ma JY (2021) MFF-GAN: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Information Fusion 66:40–53

    Article  Google Scholar 

  14. Hill P, Al-Mualla ME, Bull D (2017) Perceptual image fusion using wavelets. IEEE Trans Image Process 26(3):1076–1088

    Article  MathSciNet  MATH  Google Scholar 

  15. Petrovic VS, Xydeas CS (2004) Gradient-based multiresolution image fusion. IEEE Trans Image Process 13(2):228–237

    Article  MATH  Google Scholar 

  16. Ji X, Zhang G (2015) Image fusion method of SAR and infrared image based on curvelet transform with adaptive weighting. Multimedia Tools Appl 76(17):17633–17649

    Article  MATH  Google Scholar 

  17. Yang B, Li S (2010) Multifocus image fusion and restoration with sparse representation. IEEE Trans Instrum Meas 59(4):884–892

    Article  MATH  Google Scholar 

  18. Liu Y, Chen X, Ward RK, Wang ZJ (2016) Image fusion with convolutional sparse representation. IEEE Signal Process Lett 23(12):1882–1886

    Article  MATH  Google Scholar 

  19. Zhou Z, Li S, Wang B (2014) Multi-scale weighted gradient-based fusion for multi-focus images. Inf Fusion 20:60–72

    Article  MATH  Google Scholar 

  20. Liang J, He Y, Liu D, Zeng X (2012) Image fusion using higher order singular value decomposition. IEEE Trans Image Process 21(5):2898–2909

    Article  MathSciNet  MATH  Google Scholar 

  21. Mitianoudis N, Stathaki T (2007) Pixel-based and region-based image fusion schemes using ICA bases. Inf Fusion 8(2):131–142

    Article  MATH  Google Scholar 

  22. Aslantas V, Kurban R (2010) Fusion of multi-focus images using differential evolution algorithm. Expert Syst Appl 37:8861–8870

    Article  MATH  Google Scholar 

  23. De I, Chanda B (2013) Multi-focus image fusion using a morphologybased focus measure in a quad-tree structure. Inf Fusion 14(2):136–146

    Article  MATH  Google Scholar 

  24. Bai X, Zhang Y, Zhou F, Xue B (2015) Quadtree-based multi-focus image fusion using a weighted focus-measure. Inf Fusion 22:105–118

    Article  MATH  Google Scholar 

  25. Li S, Yang B (2008) Multifocus image fusion using region segmentation and spatial frequency. Image Vis Comput 26(7):971–979

    Article  MATH  Google Scholar 

  26. Duan J, Chen L, Chen CLP (2018) Multifocus image fusion with enhanced linear spectral clustering and fast depth map estimation. Neurocomputing 318:43–54

    Article  MATH  Google Scholar 

  27. Guo X, Nie R, Cao J, Zhou D, Qian W (2018) Fully convolutionalnetwork-based multifocus image fusion. Neural Comput 30(7):1775–1800

    Article  MathSciNet  Google Scholar 

  28. Zhao W, Wang D, Lu H (2019) Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network. IEEE Trans Circuits Syst Video Technol 29(4):1102–1115

    Article  MATH  Google Scholar 

  29. Lai R, Li Y, Guan J, Xiong A (2019) Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access 7:114385–114399

    Article  Google Scholar 

  30. Mustafa HT, Zareapoor M, Yang J (2020) MLDNet: multi-level dense network for multi-focus image fusion. Signal Process Image Commun 85:115864

  31. Zhang Y et al (2020) IFCNN: A general image fusion framework based on convolutional neural network. Inf Fusion 54:99–118

    Article  MATH  Google Scholar 

  32. Xu H, et al (2020) FusionDN: a unifified densely connected network for image fusion. Proc AAAI Conf Artifificial Intell 34(07):12484–12491

  33. Xu H, Ma J, Jiang J, Guo X, Ling H (2022) U2Fusion: a Unified Unsupervised Image Fusion Network. IEEE Trans Pattern Anal Mach Intell 44(1):502–518

    Article  MATH  Google Scholar 

  34. Ma B et al (2021) SESF-fuse: An unsupervised deep model for multi-focus image fusion. Neural Comput Appl 33(11):5793–5804

    Article  MATH  Google Scholar 

  35. Jiang LM, Fan H, Li JJ, Tu CH (2021) Pseudo-Siamese residual atrous pyramid network for multi-focus image fusion. IET Image Proc 15(13):3304–3317

    Article  MATH  Google Scholar 

  36. Wang C, Pu Y et al (2023) MCNN: conditional focus probability learning to multi-focus image fusion via mutually coupled neural network. IET Image Process 1:15

    MATH  Google Scholar 

Download references

Funding

This work is supported by the National Natural Science Foundation of China (Grant No. 52375264).

Author information

Authors and Affiliations

Authors

Contributions

Gang Hu: Conceptualization, Methodology, Validation, Writing-original draft, Writing-review & editing.

Jinlin Jiang: Methodology, Formal analysis, Writing-original draftt, Writing-review & editing.

Guanglei Sheng: Methodology, Formal analysis, Writing-original draft.

Wei Guo: Conceptualization, Validation, Supervision, Validation Declarations.

Corresponding author

Correspondence to Gang Hu.

Ethics declarations

Ethical and informed consent for data used

The dataset used is open source and can be accessed on [http://host.robots.ox.ac.uk/pascal/VOC/index.html].

Competing interests

The authors declare no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, G., Jiang, J., Sheng, G. et al. MSDCNet: Multi-stage and deep residual complementary multi-focus image fusion network based on multi-scale feature learning. Appl Intell 55, 232 (2025). https://doi.org/10.1007/s10489-024-05983-0

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05983-0

Keywords