Skip to main content
SpringerLink
Log in

Infrared and visible image fusion based on VPDE model and VGG network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Infrared (IR) and visible (VIS) image fusion techniques are widely applied to many high-level vision tasks, such as object detection, recognition, and tracking. However, most existing image fusion algorithms exhibit varying degrees of edge-step effect and texture information degradation in their fused images. To improve the fusion quality, an IR and VIS image fusion method based on a variational partial differential equation (VPDE) model and a VGG network is proposed. A productive smoothing segmentation is integrated into the energy function of the VPDE model, which is based on a novel regularization function. To decompose source images into low-frequency and high-frequency components, the new VPDE model is employed. To fuse low-frequency components, a probabilistic parameter model based on space-alternating generalized expectation-maximization (SAGE) is utilized rather than the traditional average fusion rule. Then, multi-layer features of the high-frequency components are extracted using a VGG network. To generate several candidates of the fused detail content, the \(l_1\)-norm and weighted average rule are adopted, and the final details are obtained by using the maximum selection strategy. Finally, fused images are obtained by reconstructing the fused low-frequency and high-frequency components. Extensive experiments on the TNO and RoadScene datasets demonstrate that the proposed technique effectively eliminates artifacts as well as the step effect. In the subjective comparison, the proposed method can highlight the salient objects of the fused images while strengthening the texture information. In terms of the evaluation metrics, the proposed method outperforms 13 state-of-the-art methods in objective comparison in addition to the subjective evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

The datasets produced and/or analyzed in the research at hand are available upon reasonable request to the corresponding authors.

References

  1. Ma J, Ma Y, Li C (2019) Infrared and visible image fusion methods and applications: A survey. Information Fusion 45:153–178. https://doi.org/10.1016/j.inffus.2018.02.004

    Article  Google Scholar 

  2. Hao S, He T, Ma X, An B, Wen H, Wang F (2022) Nosmfuse: An infrared and visible image fusion approach based on norm optimization and slime mold architecture. Applied Intelligence pp 1–14. https://doi.org/10.1007/s10489-022-03591-4

  3. Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L (2020) Ifcnn: A general image fusion framework based on convolutional neural network. Information Fusion 54:99–118

    Article  Google Scholar 

  4. Singh S, Mittal N, Singh H (2022) A feature level image fusion for ir and visible image using mnmra based segmentation. Neural Comput Appl 34(10):8137–8154. https://doi.org/10.1007/s00521-022-06900-7

    Article  Google Scholar 

  5. Liu J, Wu Y, Wu G, Liu R, Fan X (2022) Learn to search a lightweight architecture for target-aware infrared and visible image fusion. IEEE Signal Process Lett 29:1614–1618. https://doi.org/10.1109/LSP.2022.3180672

    Article  Google Scholar 

  6. Xu X, Liu G, Bavirisetti DP, Zhang X, Sun B, Xiao G (2022) Fast detection fusion network (fdfnet): An end to end object detection framework based on heterogeneous image fusion for power facility inspection. IEEE Trans Power Delivery 37(6):4496–4505. https://doi.org/10.1109/TPWRD.2022.3150110

    Article  Google Scholar 

  7. Zhang H, Xu H, Tian X, Jiang J, Ma J (2021) Image fusion meets deep learning: A survey and perspective. Information Fusion 76:323–336. https://doi.org/10.1016/j.inffus.2021.06.008

    Article  Google Scholar 

  8. Gao Y, Ma S, Liu J (2022) Dcdr-gan: A densely connected disentangled representation generative adversarial network for infrared and visible image fusion. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2022.3206807

    Article  Google Scholar 

  9. Li H, Wu XJ, Kittler J (2021) Rfn-nest: An end-to-end residual fusion network for infrared and visible images. Information Fusion 73:72–86. https://doi.org/10.1016/j.inffus.2021.02.023

    Article  Google Scholar 

  10. Chen J, Li X, Luo L, Mei X, Ma J (2020) Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Inf Sci 508:64–78. https://doi.org/10.1016/j.ins.2019.08.066

    Article  Google Scholar 

  11. Ding Z, Wang T, Sun Q, Wang H (2021) Adaptive fusion with multi-scale features for interactive image segmentation. Appl Intell 51(8):5610–5621. https://doi.org/10.1007/s10489-020-02114-3

    Article  Google Scholar 

  12. Hu Z, Liang W, Ding D, Wei G (2021) An improved multi-focus image fusion algorithm based on multi-scale weighted focus measure. Appl Intell 51(7):4453–4469. https://doi.org/10.1007/s10489-020-02066-8

    Article  Google Scholar 

  13. Dinh PH (2021) Multi-modal medical image fusion based on equilibrium optimizer algorithm and local energy functions. Appl Intell 51(11):8416–8431. https://doi.org/10.1007/s10489-021-02282-w

    Article  Google Scholar 

  14. Chen Y, Liu A, Liu Y, Qian R, Xie Q, Chen X (2022) Image fusion with sparse representation: A novel local contrast-based preprocessing strategy. IEEE Sensors Letters 6(5):1–4. https://doi.org/10.1109/LSENS.2022.3170744

    Article  Google Scholar 

  15. Wang C, Wu Y, Yu Y, Zhao JQ (2022) Joint patch clustering-based adaptive dictionary and sparse representation for multi-modality image fusion. Mach Vis Appl 33(5):1–16. https://doi.org/10.1007/s00138-022-01322-w

    Article  Google Scholar 

  16. Xu H, Ma J, Jiang J, Guo X, Ling H (2020) U2fusion: A unified unsupervised image fusion network. IEEE Trans Pattern Anal Mach Intell 44(1):502–518. https://doi.org/10.1109/TPAMI.2020.3012548

    Article  Google Scholar 

  17. Ma J, Xu H, Jiang J, Mei X, Zhang XP (2020) Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans Image Process 29:4980–4995. https://doi.org/10.1109/TIP.2020.2977573

    Article  MATH  Google Scholar 

  18. Li Y, Liu G, Bavirisetti DP, Gu X, Zhou X (2023) Infrared-visible image fusion method based on sparse and prior joint saliency detection and latlrr-fpde. Digital Signal Processing 134:103910. https://doi.org/10.1016/j.dsp.2023.103910

    Article  Google Scholar 

  19. Li G, Lin Y, Qu X (2021) An infrared and visible image fusion method based on multi-scale transformation and norm optimization. Information Fusion 71:109–129. https://doi.org/10.1016/j.inffus.2021.02.008

    Article  Google Scholar 

  20. Li X, Chen H, Li Y, Peng Y (2022) Mafusion: Multiscale attention network for infrared and visible image fusion. IEEE Trans Instrum Meas 71:1–16. https://doi.org/10.1109/TIM.2022.3181898

    Article  Google Scholar 

  21. Jca B, Xl A, Ll C, Xm D, Jmb D (2020) Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Inf Sci 508:64–78. https://doi.org/10.1016/j.ins.2019.08.066

    Article  Google Scholar 

  22. Singh T, Nair RR (2019) Multi sensor medical image fusion using pyramid based discrete wavelet transform:a multi-resolution approach. IET Image Proc 13(9):1447–1459. https://doi.org/10.1049/iet-ipr.2018.6556

    Article  Google Scholar 

  23. Li X, Zhou F, Tan H, Chen Y, Zuo W (2021) Multi-focus image fusion based on nonsubsampled contourlet transform and residual removal. Signal Process 184:108062. https://doi.org/10.1016/j.sigpro.2021.108062

    Article  Google Scholar 

  24. Yan H, Li Z (2020) Infrared and visual image fusion based on multi-scale feature decomposition. Optik 203:163900. https://doi.org/10.1016/j.ijleo.2019.163900

    Article  Google Scholar 

  25. Xu H, Wang X, Ma J (2021) Drf: Disentangled representation for visible and infrared image fusion. IEEE Trans Instrum Meas 70:1–13. https://doi.org/10.1109/TIM.2021.3056645

    Article  Google Scholar 

  26. Ma J, Zhang H, Shao Z, Liang P, Xu H (2020) Ganmcc: A generative adversarial network with multiclassification constraints for infrared and visible image fusion. IEEE Trans Instrum Meas 70:1–14. https://doi.org/10.1109/TIM.2020.3038013

    Article  Google Scholar 

  27. Ma J, Yu W, Liang P, Li C, Jiang J (2019) Fusiongan: A generative adversarial network for infrared and visible image fusion. Information fusion 48:11–26. https://doi.org/10.1016/j.inffus.2018.09.004

    Article  Google Scholar 

  28. Tang L, Yuan J, Ma J (2022) Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Information Fusion 82:28–42. https://doi.org/10.1016/j.inffus.2021.12.004

    Article  Google Scholar 

  29. He K, Cao X, Shi Y, Nie D, Gao Y, Shen D (2019) Pelvic organ segmentation using distinctive curve guided fully convolutional networks. IEEE Trans Med Imaging 38(2):585–595. https://doi.org/10.1109/TMI.2018.2867837

    Article  Google Scholar 

  30. Yu X, Ye X, Zhang S (2022) Floating pollutant image target extraction algorithm based on immune extremum region. Digital Signal Processing 123:103442. https://doi.org/10.1016/j.dsp.2022.103442

    Article  Google Scholar 

  31. Yu X, Tian X (2022) A fault detection algorithm for pipeline insulation layer based on immune neural network. Int J Press Vessels Pip 196:104611. https://doi.org/10.1016/j.ijpvp.2022.104611

    Article  Google Scholar 

  32. Yu X, Zhou Z, Gao Q, Li D, Rha K (2018) Infrared image segmentation using growing immune field and clone threshold. Infrared Physics & Technology 88:184–193. https://doi.org/10.1016/j.infrared.2017.11.029

    Article  Google Scholar 

  33. Yu X, Lu Y, Gao Q (2021) Pipeline image diagnosis algorithm based on neural immune ensemble learning. Int J Press Vessels Pip 189:104249. https://doi.org/10.1016/j.ijpvp.2020.104249

    Article  Google Scholar 

  34. Ngo L, Cha J, Han JH (2020) Deep neural network regression for automated retinal layer segmentation in optical coherence tomography images. IEEE Trans Image Process 29:303–312. https://doi.org/10.1109/TIP.2019.2931461

    Article  MathSciNet  MATH  Google Scholar 

  35. Ma J, Zhou Y (2020) Infrared and visible image fusion via gradientlet filter. Comput Vis Image Underst 197–198:103016. https://doi.org/10.1016/j.cviu.2020.103016

    Article  Google Scholar 

  36. Li H, Wu XJ, Kittler J (2018) Infrared and visible image fusion using a deep learning framework. In: 2018 24th international conference on pattern recognition (ICPR), IEEE, pp 2705–2710. https://doi.org/10.1109/ICPR.2018.8546006

  37. Shreyamsha Kumar B (2015) Image fusion based on pixel significance using cross bilateral filter. SIViP 9(5):1193–1204. https://doi.org/10.1007/s11760-013-0556-9

    Article  Google Scholar 

  38. Ma J, Zhou Z, Wang B, Zong H (2017) Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Physics & Technology 82:8–17. https://doi.org/10.1016/j.infrared.2017.02.005

    Article  Google Scholar 

  39. Sun P, Wang C, Li M, Liu L (2021) Partial differential equations-based iterative denoising algorithm for movie images. Adv Math Phys. https://doi.org/10.1155/2021/8176746

    Article  MathSciNet  MATH  Google Scholar 

  40. Bavirisetti DP, Dhuli R (2015) Fusion of infrared and visible sensor images based on anisotropic diffusion and karhunen-loeve transform. IEEE Sens J 16(1):203–209. https://doi.org/10.1109/JSEN.2015.2478655

    Article  Google Scholar 

  41. Bavirisetti DP, Xiao G, Liu G (2017) Multi-sensor image fusion based on fourth order partial differential equations. In: 2017 20th International conference on information fusion (Fusion), IEEE, pp 1–9. https://doi.org/10.23919/ICIF.2017.8009719

  42. Liu Y, Zhou D, Nie R, Hou R, Zhou J (2020) Robust spiking cortical model and total-variational decomposition for multimodal medical image fusion. Biomed Signal Process Control 61:101996. https://doi.org/10.1016/j.bspc.2020.101996

    Article  Google Scholar 

  43. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 https://doi.org/10.48550/arXiv.1409.1556

  44. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2414–2423

  45. Johnson KA, Becker JA (1997) The whole brain atlas. URL https://www.med.harvard.edu/AANLIB/home.htm

  46. Toet A (2014) Tno image fusion dataset. URL https://figshare.com/articles/dataset/TNO_Image_Fusion_Dataset/1008029

  47. Zhang Y, Zhang L, Bai X, Zhang L (2017) Infrared and visual image fusion through infrared feature extraction and visual information preservation. Infrared Physics & Technology 83:227–237. https://doi.org/10.1016/j.infrared.2017.05.007

    Article  Google Scholar 

  48. Li H, Wu X (2018) Infrared and visible image fusion using latent low-rank representation. CoRR abs/1804.08992. https://doi.org/10.48550/arXiv.1804.08992

  49. Li H, Jun WuX, Durrani TS (2019) Infrared and visible image fusion with resnet and zero-phase component analysis. Infrared Physics & Technology 102(103):039. https://doi.org/10.1016/j.infrared.2019.103039

    Article  Google Scholar 

  50. Tang H, Liu G, Tang L, Bavirisetti DP, Wang J (2022) Mdedfusion: A multi-level detail enhancement decomposition method for infrared and visible image fusion. Infrared Physics & Technology 127(104):435. https://doi.org/10.1016/j.infrared.2022.104435

    Article  Google Scholar 

  51. Ma J, Tang L, Fan F, Huang J, Mei X, Ma Y (2022) Swinfusion: Cross-domain long-range learning for general image fusion via swin transformer. IEEE/CAA Journal of Automatica Sinica 9(7):1200–1217. https://doi.org/10.1109/JAS.2022.105686

  52. Li C, Cheng H, Hu S, Liu X, Tang J, Lin L (2016) Learning collaborative sparse representation for grayscale-thermal tracking. IEEE Trans Image Process 25(12):5743–5756. https://doi.org/10.1109/TIP.2016.2614135

    Article  MathSciNet  MATH  Google Scholar 

  53. Li B, Yan J, Wu W, Zhu Z, Hu X (2018) High performance visual tracking with siamese region proposal network. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8971–8980. https://doi.org/10.1109/CVPR.2018.00935

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (62203224) and Shanghai Special Plan for Local Colleges and Universities for Capacity Building (22010 501300).

Author information

Authors and Affiliations

  1. Shanghai University of Electric Power, Shanghai, People’s Republic of China

    Donghua Luo, Gang Liu & Yisheng Cao

  2. Norwegian University of Science and Technology, Trondheim, Norway

    Durga Prasad Bavirisetti

Authors
  1. Donghua Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Durga Prasad Bavirisetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yisheng Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gang Liu.

Ethics declarations

Conflict of interest

We state that we do not have any personal or financial connections with individuals or entities that may improperly affect our work. We have no professional or personal interests that could be perceived as influencing the views expressed in our paper, "Infrared and visible image fusion based on VPDE model and VGG network," regarding any product, service, or company.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, D., Liu, G., Bavirisetti, D.P. et al. Infrared and visible image fusion based on VPDE model and VGG network. Appl Intell 53, 24739–24764 (2023). https://doi.org/10.1007/s10489-023-04692-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04692-4

Keywords

Search

Navigation