Skip to main content
SpringerLink
Log in

IBPNet: a multi-resolution and multi-modal image fusion network via iterative back-projection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Most multi-modal image fusion methods are based on the prerequisite that the source images have the same resolution. However, due to the limitations of the environment and hardware facilities, the resolution of multi-modal images is always distinct. For example, spatial resolution of infrared images is usually lower than that of the corresponding visible images. Therefore, our motivation is to solve the problem of blurred details or a certain degree of information loss that are prone to appear in the fusion images. Under this motivation, a novel deep learning-based multi-resolution multi-modal image fusion network via iterative back-projection (IBPNet) is constructed to get high quality fused images. The key contribution of our IBPNet is to design up-projection and down-projection blocks to realize the feature mapping conversion between high and low-resolution images. The feedback errors generated in the alternation process are self-corrected in the reconstruction process. In addition, an effective combined loss function is designed, which can adapt to different multi-resolution and multi-modal image fusion tasks. Experimental results show that our method is superior to other state-of-the-art fusion methods in terms of both visual perception and objective evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://github.com/hanna-xu/RoadScene

  2. http://www.med.harvard.edu/AANLIB/home.html

References

  1. Liu Q, Teng Q, Chen H, Li B, Qing L (2021) Dual adaptive alignment and partitioning network for visible and infrared cross-modality person re-identification. Appl Intell, 1–17. https://doi.org/10.1007/s10489-021-02390-7

  2. Fu Y, Wu XJ, Durrani T (2021) Image fusion based on generative adversarial network consistent with perception. Inform Fusion 72:110–125. https://doi.org/10.1016/j.inffus.2021.02.019

    Article  Google Scholar 

  3. Ao F, Pádua FLC, Lacerda A, Machado AC, Dalip DH (2019) Multimodal data fusion framework based on autoencoders for top-n recommender systems. Appl Intell 49(9):3267–3282. https://doi.org/10.1007/s10489-019-01430-7

    Article  Google Scholar 

  4. Li H, Li X, Yu Z, Mao C (2016) Multifocus image fusion by combining with mixed-order structure tensors and multiscale neighborhood. Inform Sci Int J 349-350:25–49. https://doi.org/10.1016/j.ins.2016.02.030

    Article  MATH  Google Scholar 

  5. Huang Y, Li W, Gao M (2018) Algebraic multi-grid based multi-focus image fusion using watershed algorithm. IEEE Access 6:47082–47091. https://doi.org/10.1109/access.2018.2866867

    Article  Google Scholar 

  6. Zhu Z, Qi G, Yi C, Chen Y (2016) A novel Multi-Focus image fusion method based on stochastic coordinate coding and local density peaks clustering. Future Internet 8(4):53. https://doi.org/10.3390/fi8040053

    Article  Google Scholar 

  7. Li L, Ma H, Jia Z, Si Y (2021) A novel multiscale transform decomposition based multi-focus image fusion framework. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-020-10462-y

  8. Jca B, Xl A, Ll C, Xm D, Jmb D (2020) Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Inf Sci 508:64–78. https://doi.org/10.1016/j.ins.2019.08.066

    Article  Google Scholar 

  9. Zhang S, Li X, Zhang X, Zhang S (2021) Infrared and visible image fusion based on saliency detection and two-scale transform decomposition. Infrared Phys Technol 114:103626. https://doi.org/10.1016/j.infrared.2020.103626

    Article  Google Scholar 

  10. Liu Y, Wang Z (2014) Simultaneous image fusion and denoising with adaptive sparse representation. Image Processing Iet 9(5):347–357. https://doi.org/10.1049/iet-ipr.2014.0311

    Article  Google Scholar 

  11. Liu Y, Chen X, Ward RK, Wang ZJ (2019) Medical image fusion via convolutional sparsity based morphological component analysis. IEEE Signal Processing Letters 26(3):485–489. https://doi.org/10.1109/LSP.2019.2895749

    Article  Google Scholar 

  12. Wei Q, Bioucas-Dias J, Dobigeon N, Tourneret J (2015) Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Trans Geosci Remote Sens 53(7):3658–3668. https://doi.org/10.1109/TGRS.2014.2381272

    Article  Google Scholar 

  13. Zhu Z, Zheng M, Qi G, Wang D, Xiang Y (2019) A phase congruency and local laplacian energy based multi-modality medical image fusion method in nsct domain. IEEE Access 7:20811–20824. https://doi.org/10.1109/ACCESS.2019.2898111

    Article  Google Scholar 

  14. Huang J, Le Z, Ma Y, Fan F, Yang L (2020) Mgmdcgan: Medical image fusion using multi-generator multi-discriminator conditional generative adversarial network. IEEE Access 8:55145–55157. https://doi.org/10.1109/ACCESS.2020.2982016

    Article  Google Scholar 

  15. Fu J, Li W, Ouyang A, He B (2021) Multimodal biomedical image fusion method via rolling guidance filter and deep convolutional neural networks. Optik - International Journal for Light and Electron Optics 237(7):166726. https://doi.org/10.1016/j.ijleo.2021.166726

    Article  Google Scholar 

  16. Li H, Zhang L, Jiang M, Li Y (2020) Multi-focus image fusion algorithm based on supervised learning for fully convolutionalneural networks. Laser Optoelectronics Progress 57(8):081015. https://doi.org/10.3788/LOP57.081015

    Article  Google Scholar 

  17. Li H, Wu XJ, Durrani TS (2019) Infrared and visible image fusion with ResNet and zero-phase component analysis. Infrared Physics Technology 102:103039. https://doi.org/10.1016/j.infrared.2019.103039

    Article  Google Scholar 

  18. Ma J, Wei Y, Liang P, Chang L, Jiang J (2019) Fusiongan:A a generative adversarial network for infrared and visible image fusion. 48:11–26. https://doi.org/10.1016/j.inffus.2018.09.004

  19. Ma J, Xu H, Jiang J, Mei X, Zhang XP (2020) Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans Image Process 29:4980–4995. https://doi.org/10.1109/TIP.2020.2977573

    Article  MATH  Google Scholar 

  20. Du Q, Xu H, Ma Y, Huang J, Fan F (2018) Fusing infrared and visible images of different resolutions via total variation model. Sensors 18(11):3827. https://doi.org/10.3390/s18113827

    Article  Google Scholar 

  21. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 2472-2481. https://doi.org/10.1109/CVPR.2018.00262

  22. Zhang H, Xu H, Xiao Y, Guo X, Ma J (2020) Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity. Proceedings of the AAAI Conference on Artificial Intelligence 34(7):12797–12804. https://doi.org/10.1609/aaai.v34i07.6975

    Article  Google Scholar 

  23. Irani M, Peleg S (1991) Improving resolution by image registration. GVGIP : Graphical Models and Image Processing 53(3):231–239. https://doi.org/10.1016/1049-9652(91)90045-L

    Google Scholar 

  24. Dai S, Mei H, Ying W, Gong Y (2008) Bilateral back-projection for single image super resolution. Multimedia and Expo, 2007 IEEE International Conference on IEEE, 1039–1042. https://doi.org/10.1109/ICME.2007.4284831

  25. Dong W, Lei Z, Shi G, Wu X (2010) Nonlocal back-projection for adaptive image enlargement. IEEE International Conference on Image Processing, 349–352. https://doi.org/10.1109/ICIP.2009.5414423

  26. Haris M, Shakhnarovich G, Ukita N (2018) Deep Back-Projection Networks For Super-Resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition 1664–1673. https://doi.org/10.1109/CVPR.2018.00179

  27. Hou R, Zhou D, Nie R, Liu D, Xiong L, Guo L, Yu C (2020). In: IEEE Transactions on Computational Imaging. https://doi.org/10.1109/TCI.2020.2965304, vol 6, pp 640–651

  28. Dinh PH (2021) Multi-modal medical image fusion based on equilibrium optimizer algorithm and local energy functions. Applied Intelligence. https://doi.org/10.1007/s10489-021-02282-w

  29. Ma J, Chen C, Li C, Huang J (2016) Infrared and visible image fusion via gradient transfer and total variation minimization. Inform Fusion 31:100–109. https://doi.org/10.1016/j.inffus.2016.02.001

    Article  Google Scholar 

  30. Li H, Wu X (2018) Densefuse: A fusion approach to infrared and visible images. IEEE Trans Image Process 28(5):2614–2623. https://doi.org/10.1109/TIP.2018.2887342

    Article  MathSciNet  Google Scholar 

  31. Aardt V (2008) Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing 2(1):1–28. https://doi.org/10.1117/1.2945910

    Google Scholar 

  32. Eskicioglu AM, Fisher PS (1995) Image quality measures and their performance. IEEE Trans Commun 43(12):2959–2965. https://doi.org/10.1109/26.477498

    Article  Google Scholar 

  33. Han Y, Cai Y, Cao Y, Xu X (2013) A new image fusion performance metric based on visual information fidelity. Information Fusion 14(2):127–135. https://doi.org/10.1016/j.inffus.2011.08.002

    Article  Google Scholar 

  34. Piella G, Heijmans H (2003) A new quality metric for image fusion. International Conference on Image Processing 3:III–173. https://doi.org/2003.10.1109/ICIP.2003.1247209

    Article  Google Scholar 

  35. Wei T, Tiwari P, Pandey HM, Moreira C, Jaiswal AK (2020) Multimodal medical image fusion algorithm in the era of big data. Neural Computing and Applications (3):. https://doi.org/10.1007/s00521-020-05173-2

  36. Lahoud F, Süsstrunk S (2020) Zero-learning fast medical image fusion. In: 2019 22th International Conference on Information Fusion (FUSION)

  37. Xydeas CS, Pv V (2000) Objective image fusion performance measure. Military Technical Courier 56(4):181–193. https://doi.org/10.5937/vojtehg0802181B

    Google Scholar 

  38. Zhou W, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9 (3):81–84. https://doi.org/10.1109/97.995823

    Article  Google Scholar 

Download references

Acknowledgments

This paper is supported by the National Natural Science Foundation of China (Nos.61871210, 62071213, 61901209), Chuanshan Talent Project of the University of South China, the construct program of key disciplines in USC (No. NHXK04), and Scientific Research Fund of Hengyang Science and Technology Bureau (Nos. 2015KG51).

Author information

Authors and Affiliations

  1. University of South China, Hengyang, 421001, China

    Chang Liu, Bin Yang, Xiaozhi Zhang & Lihui Pang

Authors
  1. Chang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaozhi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lihui Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bin Yang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, C., Yang, B., Zhang, X. et al. IBPNet: a multi-resolution and multi-modal image fusion network via iterative back-projection. Appl Intell 52, 16185–16201 (2022). https://doi.org/10.1007/s10489-022-03375-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03375-w

Keywords

Search

Navigation