Abstract
Addressing the boundary blurring problem in focus and out-of-focus regions is a key area of research in multifocus image fusion. Effective utilization of multi-scale modules is essential for enhancing performance. Therefore, this paper proposes a multi-stage feature extraction and deep residual complementary multifocus image fusion network. In the feature extraction stage, the V-shaped connection module captures the main objects and contours of the image. The feature thinning extraction module uses extended convolution to learn image details and refine textures at multiple scales. The advanced feature texture enhancement module targets boundary blurring regions, enhancing texture details and improving fusion quality. Asymmetric convolution reduces the network’s computational burden, improving feature learning efficiency. The fusion strategy uses a compound loss function to ensure image quality and prevent color distortion. The image reconstruction module uses residual connections with different-sized convolution kernels to maintain feature consistency and improve image quality. The network utilizes a dual-path Pseudo-Siamese structure, which handles image focus and defocus regions separately. Experimental results demonstrate the algorithm’s effectiveness. On the Lytro dataset, it achieves AG and EI metric values of 6.9 and 72.5, respectively, outperforming other methods. Fusion metrics SD = 61.80, SF = 19.63, and VIF = 0.94 surpass existing algorithms, effectively resolving the boundary blurring problem and providing better visual perception and broader applicability.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The datasets generated during and analysed during the current study are available from the corresponding author on reasonable request.
References
Li H, Wu XJ, Kittler J (2020) MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Trans Image Process 29:4733–4746
Yang P, Gao LF, Zi LL (2021) “Image fusion method of convolution sparsity and detail saliency map analysis. Journal of Image and Graphics 26(10):2433–2449
Huo X, Zou Y, Chen Y, Tan JQ (2021) Dual-scale decomposition and saliency analysis based infrared and visible image fusion. Journal of Image and Graphics 26(12):2813–2825
Li S, Kwok JT, Wang Y (2001) ‘Combination of images with diverse focuses using the spatial frequency.’ Inf Fusion 2(3):169–176
Li S, Kwok JT, Wang Y (2002) ‘Multifocus image fusion using artifificial neural networks.’ Pattern Recognit Lett 23(8):985–997
Geng T, Liu XY, Wang X et al (2021) Deep shearlet residual learning network for single image super-resolution. IEEE Trans Image Process 30:4129–4142
Liu Y, Liu S, Wang Z (2015) Multi-focus image fusion with dense SIFT’. Inf Fusion 23:139–155
Tang H, Xiao B, Li W, Wang G (2018) Pixel convolutional neural network for multi-focus image fusion’. Inf Sci 433–434:125–141
Amin-Naji M, Aghagolzadeh A, Ezoji M (2019) Ensemble of CNN for multi-focus image fusion’. Inf Fusion 51:201–214
Amin-Naji M, Aghagolzadeh A, Ezoji M (2020) CNNs hard voting for multi-focus image fusion[J]. J Ambient Intell Humanized Comput 11(4):1749–1769
Ma JY, Tang LF, Xu ML, Zhang H, Xiao GB (2021) STDFusionNet: an infrared and visible image fusion network based on salient target detection. IEEE Trans Instrum Meas 70:1–13
Wang YC, Xu S, Liu JM, Zhao ZX, Zhang CX, Zhang JS (2021) MFIF-GAN: a new generative adversarial network for multifocus image fusion. Signal Processing: Image Communication 96:116295
Zhang H, Le ZL, Shao ZF, Xu H, Ma JY (2021) MFF-GAN: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Information Fusion 66:40–53
Hill P, Al-Mualla ME, Bull D (2017) Perceptual image fusion using wavelets. IEEE Trans Image Process 26(3):1076–1088
Petrovic VS, Xydeas CS (2004) Gradient-based multiresolution image fusion. IEEE Trans Image Process 13(2):228–237
Ji X, Zhang G (2015) Image fusion method of SAR and infrared image based on curvelet transform with adaptive weighting. Multimedia Tools Appl 76(17):17633–17649
Yang B, Li S (2010) Multifocus image fusion and restoration with sparse representation. IEEE Trans Instrum Meas 59(4):884–892
Liu Y, Chen X, Ward RK, Wang ZJ (2016) Image fusion with convolutional sparse representation. IEEE Signal Process Lett 23(12):1882–1886
Zhou Z, Li S, Wang B (2014) Multi-scale weighted gradient-based fusion for multi-focus images. Inf Fusion 20:60–72
Liang J, He Y, Liu D, Zeng X (2012) Image fusion using higher order singular value decomposition. IEEE Trans Image Process 21(5):2898–2909
Mitianoudis N, Stathaki T (2007) Pixel-based and region-based image fusion schemes using ICA bases. Inf Fusion 8(2):131–142
Aslantas V, Kurban R (2010) Fusion of multi-focus images using differential evolution algorithm. Expert Syst Appl 37:8861–8870
De I, Chanda B (2013) Multi-focus image fusion using a morphologybased focus measure in a quad-tree structure. Inf Fusion 14(2):136–146
Bai X, Zhang Y, Zhou F, Xue B (2015) Quadtree-based multi-focus image fusion using a weighted focus-measure. Inf Fusion 22:105–118
Li S, Yang B (2008) Multifocus image fusion using region segmentation and spatial frequency. Image Vis Comput 26(7):971–979
Duan J, Chen L, Chen CLP (2018) Multifocus image fusion with enhanced linear spectral clustering and fast depth map estimation. Neurocomputing 318:43–54
Guo X, Nie R, Cao J, Zhou D, Qian W (2018) Fully convolutionalnetwork-based multifocus image fusion. Neural Comput 30(7):1775–1800
Zhao W, Wang D, Lu H (2019) Multi-focus image fusion with a natural enhancement via a joint multi-level deeply supervised convolutional neural network. IEEE Trans Circuits Syst Video Technol 29(4):1102–1115
Lai R, Li Y, Guan J, Xiong A (2019) Multi-scale visual attention deep convolutional neural network for multi-focus image fusion. IEEE Access 7:114385–114399
Mustafa HT, Zareapoor M, Yang J (2020) MLDNet: multi-level dense network for multi-focus image fusion. Signal Process Image Commun 85:115864
Zhang Y et al (2020) IFCNN: A general image fusion framework based on convolutional neural network. Inf Fusion 54:99–118
Xu H, et al (2020) FusionDN: a unifified densely connected network for image fusion. Proc AAAI Conf Artifificial Intell 34(07):12484–12491
Xu H, Ma J, Jiang J, Guo X, Ling H (2022) U2Fusion: a Unified Unsupervised Image Fusion Network. IEEE Trans Pattern Anal Mach Intell 44(1):502–518
Ma B et al (2021) SESF-fuse: An unsupervised deep model for multi-focus image fusion. Neural Comput Appl 33(11):5793–5804
Jiang LM, Fan H, Li JJ, Tu CH (2021) Pseudo-Siamese residual atrous pyramid network for multi-focus image fusion. IET Image Proc 15(13):3304–3317
Wang C, Pu Y et al (2023) MCNN: conditional focus probability learning to multi-focus image fusion via mutually coupled neural network. IET Image Process 1:15
Funding
This work is supported by the National Natural Science Foundation of China (Grant No. 52375264).
Author information
Authors and Affiliations
Contributions
Gang Hu: Conceptualization, Methodology, Validation, Writing-original draft, Writing-review & editing.
Jinlin Jiang: Methodology, Formal analysis, Writing-original draftt, Writing-review & editing.
Guanglei Sheng: Methodology, Formal analysis, Writing-original draft.
Wei Guo: Conceptualization, Validation, Supervision, Validation Declarations.
Corresponding author
Ethics declarations
Ethical and informed consent for data used
The dataset used is open source and can be accessed on [http://host.robots.ox.ac.uk/pascal/VOC/index.html].
Competing interests
The authors declare no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hu, G., Jiang, J., Sheng, G. et al. MSDCNet: Multi-stage and deep residual complementary multi-focus image fusion network based on multi-scale feature learning. Appl Intell 55, 232 (2025). https://doi.org/10.1007/s10489-024-05983-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-05983-0