Abstract
Bright-dark components and edge details are the most important complementary information between infrared and visible images. To extract and fuse them efficiently, a novel non-subsampled morphological fusion algorithm is proposed in this paper. The algorithm uses non-subsampled pyramid (NSP) as the spatial-frequency splitter to decompose the source image to get a series of high-frequency detail images and one low-frequency background image. Then, a dual-channel multi-scale top–bottom hat (MTBH) decomposition is constructed to extract the bright-dark details from the low-frequency background. In addition, to extract the edge details with different directions from high-frequency images, a dual-channel multidirectional inner-outer edge (MIOE) decomposition is constructed. Through these decompositions, the bright-dark information and edge details present in the source images can be effectively extracted. Then, based on the distinct roles of the extracted information, the decomposed images are fused using diverse fusion strategies. Subsequently, the fused image is reconstructed using the appropriate inverse transforms corresponding to each decomposition. The experimental results demonstrate that the fusion images generated by this algorithm exhibit richer details and higher image contrast compared to those produced by state-of-the-art algorithms.












Similar content being viewed by others
Data availability
All experimental images can be obtained through the public dataset “Toet A. TNO Image fusion dataset” https://figshare.com/articles/TN_Image_Fusion_Dataset/1008029. In addition, the experimental fusion images will be made available upon reasonable request for academic use and within the limitations of the provided informed consent by the corresponding author upon acceptance.
References
M. Arif, G. Wang, Fast curvelet transform through genetic algorithm for multimodal medical image fusion. Soft. Comput. 24(3), 1815–1836 (2020)
A. Averbuch, P. Neittaanmäki, V. Zheludev, M. Salhov, J. Hauser, Image inpainting using directional wavelet packets originating from polynomial splines. Signal Process. Image Commun.cation 97, 116334 (2021)
A. Averbuch, P. Neittaanmäki, V. Zheludev, M. Salhov, J. Hauser, Coupling BM3D with directional wavelet packets for image Denoising. arXiv preprint arXiv:2008.11595 (2020)
R.H. Bamberger, M.J. Smith, A filter bank for the directional decomposition of images: theory and design. IEEE Trans. Signal Process. 40(4), 882–893 (1992)
D. P. Bavirisetti, G. Xiao, G. Liu, Multi-sensor image fusion based on fourth order partial differential equations. In 2017 20th International conference on information fusion (Fusion) (2017). pp. 1–9
E. Candes, L. Demanet, D. Donoho, L. Ying, Fast discrete curvelet transforms. Multisc. Model. Simulat. 5(3), 861–899 (2006)
J. Chen, X. Li, L. Luo, X. Mei, J. Ma, Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Inf. Sci. 508, 64–78 (2020)
B. Cheng, L. Jin, G. Li, Infrared and visual image fusion using LNSST and an adaptive dual-channel PCNN with triple-linking strength. Neurocomputing 310, 135–147 (2018)
T. Deepika, Analysis and comparison of different wavelet transform methods using benchmarks for image fusion. arXiv preprint arXiv:2007.11488 (2020)
Y. Han, Y. Cai, Y. Cao, X. Xu, A new image fusion performance metric based on visual information fidelity. Information Fusion 14(2), 127–135 (2013)
P. Hu, F. Yang, H. Wei, L. Ji, X. Wang, Research on constructing difference-features to guide the fusion of dual-modal infrared images. Infrared Phys. Technol. 102, 102994 (2019)
P. Hu, F. Yang, L. Ji, Z. Li, H. Wei, An efficient fusion algorithm based on hybrid multiscale decomposition for infrared-visible and multi-type images. Infrared Phys. Technol. 112, 103601 (2021)
M. Kumar, N. Ranjan, B. Chourasia, Hybrid methods of contourlet transform and particle swarm optimization for multimodal medical image fusion. In 2021 international conference on artificial intelligence and smart systems (ICAIS) (2021), pp. 945–951
G. Kutyniok, D. Labate, Construction of regular and irregular shearlet frames. J. Wavelet Theory Appl 1(1), 1–12 (2007)
H.J. Kwon, S.H. Lee, Visible and near-infrared image acquisition and fusion for night surveillance. Chemosensors 9(4), 75 (2021)
S. Li, X. Kang, L. Fang, J. Hu, H. Yin, Pixel-level image fusion: a survey of the state of the art. Inform. Fusion 33, 100–112 (2017)
H. Li, L. Liu, W. Huang, C. Yue, An improved fusion algorithm for infrared and visible images based on multi-scale transform. Infrared Phys. Technol. 74, 28–37 (2016)
H. Li, X. J. Wu, J. Kittler, Infrared and visible image fusion using a deep learning framework. In 2018 24th international conference on pattern recognition (ICPR) (2018), pp. 2705–2710
H. Li, X.J. Wu, DenseFuse: a fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2018)
H. Li, X.J. Wu, T.S. Durrani, Infrared and visible image fusion with ResNet and zero-phase component analysis. Infrared Phys. Technol. 102, 103039 (2019)
J. Li, H. Huo, C. Li, R. Wang, Q. Feng, AttentionFGAN: Infrared and visible image fusion using attention-based generative adversarial networks. IEEE Trans. Multimedia 23, 1383–1396 (2020)
H. Li, X. J. Wu, Infrared and visible image fusion using latent low-rank representation. arXiv preprint arXiv:1804.08992 (2018)
H. Liu, G.F. Xiao, Y.L. Tan, C.J. Ouyang, Multi-source remote sensing image registration based on contourlet transform and multiple feature fusion. Int. J. Autom. Comput. 16, 575–588 (2019)
C.H. Liu, Y. Qi, W.R. Ding, Infrared and visible image fusion method based on saliency detection in sparse domain. Infrared Phys. Technol. 83, 94–102 (2017)
J. Liu, K. Tian, H. Xiong, Y. Zheng, Fast denoising of multi-channel transcranial magnetic stimulation signal based on improved generalized mathematical morphological filtering. Biomed. Signal Process. Control 72, 103348 (2022)
J. Ma, Y. Ma, C. Li, Infrared and visible image fusion methods and applications: a survey. Information fusion 45, 153–178 (2019)
J. Ma, W. Yu, P. Liang, C. Li, J. Jiang, FusionGAN: A generative adversarial network for infrared and visible image fusion. Inform Fusion 48, 11–26 (2019)
J. Ma, Z. Zhou, B. Wang, H. Zong, Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys. Technol. 82, 8–17 (2017)
S. Minghui, L. Lu, P. Yuanxi, J. Tian, L. Jun, Infrared & visible images fusion based on redundant directional lifting-based wavelet and saliency detection. Infrared Phys. Technol. 101, 45–55 (2019)
V.P.S. Naidu, Image fusion technique using multi-resolution singular value decomposition. Def. Sci. J. 61(5), 479 (2011)
G. Piella, H. Heijmans, A new quality metric for image fusion. In proceedings 2003 international conference on image processing (Cat. No. 03CH37429) (2003) Vol. 3, pp. III-173
G. Qi, M. Zheng, Z. Zhu, R. Yuan, A DT-CWT-based infrared-visible image fusion method for smart city. Int. J. Simul. Process Model. 14(6), 559–570 (2019)
G. Qu, D. Zhang, P. Yan, Information measure for performance of image fusion. Electron. Lett. 38(7), 1 (2002)
J.W. Roberts, J.A. Van Aardt, F.B. Ahmed, Assessment of image fusion procedures using entropy, image quality, and multispectral classification. J. Appl. Remote Sens. 2(1), 023522 (2008)
X. Wang, K. Zhang, J. Yan, M. Xing, D. Yang, Infrared image complexity metric for automatic target recognition based on neural network and traditional approach fusion. Arab. J. Sci. Eng. 45, 3245–3255 (2020)
Z. Wang, A.C. Bovik, A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)
H. Xu, D. Xu, S. Chen, W. Ma, Z. Shi, Rapid determination of soil class based on visible-near infrared, mid-infrared spectroscopy and data fusion. Remote Sens. 12(9), 1512 (2020)
C.S. Xydeas, V. Petrovic, Objective image fusion performance measure. Electron. Lett. 36(4), 308–309 (2000)
X. Zhang, P. Ye, G. Xiao, VIFB: A visible and infrared image fusion benchmark. In proceedings of the IEEE/CVF Conference on computer vision and pattern recognition workshops (2020) pp. 104–105
C. Zhao, Y. Huang, Infrared and visible image fusion method based on rolling guidance filter and NSST. Int. J. Wavelets Multiresolut. Inf. Process. 17(06), 1950045 (2019)
J. Zhou, W. Li, P. Zhang, J. Luo, S. Li, J. Zhao, Infrared and visible image fusion method based on NSST and guided filtering. Optoelectr. Sci. Mater. 11606, 82–88 (2020)
Z. Zhou, B. Wang, S. Li, M. Dong, Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters. Inform. Fusion 30, 15–26 (2016)
J. Zhou, A. L. Cunha, M. N. Do, Nonsubsampled contourlet transform: construction and application in enhancement. In IEEE international conference on image processing 2005 (2005) 1: I-469
P. Zhu, Z. Huang, A fusion method for infrared–visible image and infrared-polarization image based on multi-scale center-surround top-hat transform. Opt. Rev. 24, 370–382 (2017)
K. Zhuo, Y. HaiTao, Z. FengJie, L. Yang, J. Qi, W. JinYu, Research on multi-focal image fusion based on wavelet transform. J. Phys. Conf. Ser. 1994(1), 012018 (2021)
Acknowledgements
We sincerely thank the reviewers and editors for carefully checking our manuscript. This work is supported by the Scientific Research Foundation of the Education Department of Anhui Province (No. 2022AH050801), Scientific Research Fund for Young Teachers of Anhui University of Science and Technology (No. QNZD2021-02), Anhui Provincial Natural Science Foundation (No. 2208085ME128), Scientific Research Fund of Anhui University of Science and Technology (No. 13210679), Huainan Science and Technology Planning Project (No. 2021005).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hu, P., Wang, C., Li, D. et al. NSMT: A Novel Non-subsampled Morphological Transform Fusion Algorithm for Infrared–Visible Images. Circuits Syst Signal Process 43, 1298–1318 (2024). https://doi.org/10.1007/s00034-023-02523-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00034-023-02523-y