Skip to main content
SpringerLink
Log in

NSMT: A Novel Non-subsampled Morphological Transform Fusion Algorithm for Infrared–Visible Images

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

Bright-dark components and edge details are the most important complementary information between infrared and visible images. To extract and fuse them efficiently, a novel non-subsampled morphological fusion algorithm is proposed in this paper. The algorithm uses non-subsampled pyramid (NSP) as the spatial-frequency splitter to decompose the source image to get a series of high-frequency detail images and one low-frequency background image. Then, a dual-channel multi-scale top–bottom hat (MTBH) decomposition is constructed to extract the bright-dark details from the low-frequency background. In addition, to extract the edge details with different directions from high-frequency images, a dual-channel multidirectional inner-outer edge (MIOE) decomposition is constructed. Through these decompositions, the bright-dark information and edge details present in the source images can be effectively extracted. Then, based on the distinct roles of the extracted information, the decomposed images are fused using diverse fusion strategies. Subsequently, the fused image is reconstructed using the appropriate inverse transforms corresponding to each decomposition. The experimental results demonstrate that the fusion images generated by this algorithm exhibit richer details and higher image contrast compared to those produced by state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

All experimental images can be obtained through the public dataset “Toet A. TNO Image fusion dataset” https://figshare.com/articles/TN_Image_Fusion_Dataset/1008029. In addition, the experimental fusion images will be made available upon reasonable request for academic use and within the limitations of the provided informed consent by the corresponding author upon acceptance.

References

  1. M. Arif, G. Wang, Fast curvelet transform through genetic algorithm for multimodal medical image fusion. Soft. Comput. 24(3), 1815–1836 (2020)

    Article  Google Scholar 

  2. A. Averbuch, P. Neittaanmäki, V. Zheludev, M. Salhov, J. Hauser, Image inpainting using directional wavelet packets originating from polynomial splines. Signal Process. Image Commun.cation 97, 116334 (2021)

    Article  Google Scholar 

  3. A. Averbuch, P. Neittaanmäki, V. Zheludev, M. Salhov, J. Hauser, Coupling BM3D with directional wavelet packets for image Denoising. arXiv preprint arXiv:2008.11595 (2020)

  4. R.H. Bamberger, M.J. Smith, A filter bank for the directional decomposition of images: theory and design. IEEE Trans. Signal Process. 40(4), 882–893 (1992)

    Article  Google Scholar 

  5. D. P. Bavirisetti, G. Xiao, G. Liu, Multi-sensor image fusion based on fourth order partial differential equations. In 2017 20th International conference on information fusion (Fusion) (2017). pp. 1–9

  6. E. Candes, L. Demanet, D. Donoho, L. Ying, Fast discrete curvelet transforms. Multisc. Model. Simulat. 5(3), 861–899 (2006)

    Article  MathSciNet  Google Scholar 

  7. J. Chen, X. Li, L. Luo, X. Mei, J. Ma, Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Inf. Sci. 508, 64–78 (2020)

    Article  Google Scholar 

  8. B. Cheng, L. Jin, G. Li, Infrared and visual image fusion using LNSST and an adaptive dual-channel PCNN with triple-linking strength. Neurocomputing 310, 135–147 (2018)

    Article  Google Scholar 

  9. T. Deepika, Analysis and comparison of different wavelet transform methods using benchmarks for image fusion. arXiv preprint arXiv:2007.11488 (2020)

  10. Y. Han, Y. Cai, Y. Cao, X. Xu, A new image fusion performance metric based on visual information fidelity. Information Fusion 14(2), 127–135 (2013)

    Article  Google Scholar 

  11. P. Hu, F. Yang, H. Wei, L. Ji, X. Wang, Research on constructing difference-features to guide the fusion of dual-modal infrared images. Infrared Phys. Technol. 102, 102994 (2019)

    Article  Google Scholar 

  12. P. Hu, F. Yang, L. Ji, Z. Li, H. Wei, An efficient fusion algorithm based on hybrid multiscale decomposition for infrared-visible and multi-type images. Infrared Phys. Technol. 112, 103601 (2021)

    Article  Google Scholar 

  13. M. Kumar, N. Ranjan, B. Chourasia, Hybrid methods of contourlet transform and particle swarm optimization for multimodal medical image fusion. In 2021 international conference on artificial intelligence and smart systems (ICAIS) (2021), pp. 945–951

  14. G. Kutyniok, D. Labate, Construction of regular and irregular shearlet frames. J. Wavelet Theory Appl 1(1), 1–12 (2007)

    Google Scholar 

  15. H.J. Kwon, S.H. Lee, Visible and near-infrared image acquisition and fusion for night surveillance. Chemosensors 9(4), 75 (2021)

    Article  Google Scholar 

  16. S. Li, X. Kang, L. Fang, J. Hu, H. Yin, Pixel-level image fusion: a survey of the state of the art. Inform. Fusion 33, 100–112 (2017)

    Article  Google Scholar 

  17. H. Li, L. Liu, W. Huang, C. Yue, An improved fusion algorithm for infrared and visible images based on multi-scale transform. Infrared Phys. Technol. 74, 28–37 (2016)

    Article  Google Scholar 

  18. H. Li, X. J. Wu, J. Kittler, Infrared and visible image fusion using a deep learning framework. In 2018 24th international conference on pattern recognition (ICPR) (2018), pp. 2705–2710

  19. H. Li, X.J. Wu, DenseFuse: a fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2018)

    Article  MathSciNet  Google Scholar 

  20. H. Li, X.J. Wu, T.S. Durrani, Infrared and visible image fusion with ResNet and zero-phase component analysis. Infrared Phys. Technol. 102, 103039 (2019)

    Article  Google Scholar 

  21. J. Li, H. Huo, C. Li, R. Wang, Q. Feng, AttentionFGAN: Infrared and visible image fusion using attention-based generative adversarial networks. IEEE Trans. Multimedia 23, 1383–1396 (2020)

    Article  Google Scholar 

  22. H. Li, X. J. Wu, Infrared and visible image fusion using latent low-rank representation. arXiv preprint arXiv:1804.08992 (2018)

  23. H. Liu, G.F. Xiao, Y.L. Tan, C.J. Ouyang, Multi-source remote sensing image registration based on contourlet transform and multiple feature fusion. Int. J. Autom. Comput. 16, 575–588 (2019)

    Article  Google Scholar 

  24. C.H. Liu, Y. Qi, W.R. Ding, Infrared and visible image fusion method based on saliency detection in sparse domain. Infrared Phys. Technol. 83, 94–102 (2017)

    Article  Google Scholar 

  25. J. Liu, K. Tian, H. Xiong, Y. Zheng, Fast denoising of multi-channel transcranial magnetic stimulation signal based on improved generalized mathematical morphological filtering. Biomed. Signal Process. Control 72, 103348 (2022)

    Article  Google Scholar 

  26. J. Ma, Y. Ma, C. Li, Infrared and visible image fusion methods and applications: a survey. Information fusion 45, 153–178 (2019)

    Article  Google Scholar 

  27. J. Ma, W. Yu, P. Liang, C. Li, J. Jiang, FusionGAN: A generative adversarial network for infrared and visible image fusion. Inform Fusion 48, 11–26 (2019)

    Article  Google Scholar 

  28. J. Ma, Z. Zhou, B. Wang, H. Zong, Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys. Technol. 82, 8–17 (2017)

    Article  Google Scholar 

  29. S. Minghui, L. Lu, P. Yuanxi, J. Tian, L. Jun, Infrared & visible images fusion based on redundant directional lifting-based wavelet and saliency detection. Infrared Phys. Technol. 101, 45–55 (2019)

    Article  Google Scholar 

  30. V.P.S. Naidu, Image fusion technique using multi-resolution singular value decomposition. Def. Sci. J. 61(5), 479 (2011)

    Article  MathSciNet  Google Scholar 

  31. G. Piella, H. Heijmans, A new quality metric for image fusion. In proceedings 2003 international conference on image processing (Cat. No. 03CH37429) (2003) Vol. 3, pp. III-173

  32. G. Qi, M. Zheng, Z. Zhu, R. Yuan, A DT-CWT-based infrared-visible image fusion method for smart city. Int. J. Simul. Process Model. 14(6), 559–570 (2019)

    Article  Google Scholar 

  33. G. Qu, D. Zhang, P. Yan, Information measure for performance of image fusion. Electron. Lett. 38(7), 1 (2002)

    Article  Google Scholar 

  34. J.W. Roberts, J.A. Van Aardt, F.B. Ahmed, Assessment of image fusion procedures using entropy, image quality, and multispectral classification. J. Appl. Remote Sens. 2(1), 023522 (2008)

    Article  Google Scholar 

  35. X. Wang, K. Zhang, J. Yan, M. Xing, D. Yang, Infrared image complexity metric for automatic target recognition based on neural network and traditional approach fusion. Arab. J. Sci. Eng. 45, 3245–3255 (2020)

    Article  Google Scholar 

  36. Z. Wang, A.C. Bovik, A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)

    Article  Google Scholar 

  37. H. Xu, D. Xu, S. Chen, W. Ma, Z. Shi, Rapid determination of soil class based on visible-near infrared, mid-infrared spectroscopy and data fusion. Remote Sens. 12(9), 1512 (2020)

    Article  Google Scholar 

  38. C.S. Xydeas, V. Petrovic, Objective image fusion performance measure. Electron. Lett. 36(4), 308–309 (2000)

    Article  Google Scholar 

  39. X. Zhang, P. Ye, G. Xiao, VIFB: A visible and infrared image fusion benchmark. In proceedings of the IEEE/CVF Conference on computer vision and pattern recognition workshops (2020) pp. 104–105

  40. C. Zhao, Y. Huang, Infrared and visible image fusion method based on rolling guidance filter and NSST. Int. J. Wavelets Multiresolut. Inf. Process. 17(06), 1950045 (2019)

    Article  MathSciNet  Google Scholar 

  41. J. Zhou, W. Li, P. Zhang, J. Luo, S. Li, J. Zhao, Infrared and visible image fusion method based on NSST and guided filtering. Optoelectr. Sci. Mater. 11606, 82–88 (2020)

    Google Scholar 

  42. Z. Zhou, B. Wang, S. Li, M. Dong, Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters. Inform. Fusion 30, 15–26 (2016)

    Article  Google Scholar 

  43. J. Zhou, A. L. Cunha, M. N. Do, Nonsubsampled contourlet transform: construction and application in enhancement. In IEEE international conference on image processing 2005 (2005) 1: I-469

  44. P. Zhu, Z. Huang, A fusion method for infrared–visible image and infrared-polarization image based on multi-scale center-surround top-hat transform. Opt. Rev. 24, 370–382 (2017)

    Article  Google Scholar 

  45. K. Zhuo, Y. HaiTao, Z. FengJie, L. Yang, J. Qi, W. JinYu, Research on multi-focal image fusion based on wavelet transform. J. Phys. Conf. Ser. 1994(1), 012018 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

We sincerely thank the reviewers and editors for carefully checking our manuscript. This work is supported by the Scientific Research Foundation of the Education Department of Anhui Province (No. 2022AH050801), Scientific Research Fund for Young Teachers of Anhui University of Science and Technology (No. QNZD2021-02), Anhui Provincial Natural Science Foundation (No. 2208085ME128), Scientific Research Fund of Anhui University of Science and Technology (No. 13210679), Huainan Science and Technology Planning Project (No. 2021005).

Author information

Authors and Affiliations

  1. State Key Laboratory of Mining Response and Disaster Prevention and Control in Deep Coal Mines, Anhui University of Science and Technology, Huainan, 232001, China

    Peng Hu, Chenjun Wang, Dequan Li & Xin Zhao

  2. School of Artificial Intelligence, Anhui University of Science and Technology, Huainan, 232001, China

    Peng Hu, Chenjun Wang, Dequan Li & Xin Zhao

Authors
  1. Peng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenjun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dequan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peng Hu.

Ethics declarations

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, P., Wang, C., Li, D. et al. NSMT: A Novel Non-subsampled Morphological Transform Fusion Algorithm for Infrared–Visible Images. Circuits Syst Signal Process 43, 1298–1318 (2024). https://doi.org/10.1007/s00034-023-02523-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-023-02523-y

Keywords

Search

Navigation