Skip to main content
Springer Nature Link
Log in

VMDM-fusion: a saliency feature representation method for infrared and visible image fusion

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

A VGG-based fusion method (named VMDM-Fusion) that employs multiple decision maps is proposed to fuse infrared and visible images. Our method first feeds the infrared and visible images into a pre-trained model of VGG-16 to extract the features. Then, a feature representation method we designed uses these features to construct saliency maps. Next, these maps, in combination with a guided filter, are used to construct multiple decision maps. Lastly, the final fused image is obtained by weighting the source images based on the multiple decision maps. This is the first time a decision map is introduced in the field of infrared and visible image fusion. The experimental results demonstrate that the proposed method outperforms state-of-the-art infrared and visible image fusion methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Li, S., Kang, X., Fang, L., J, Hu., Yin, H.: Pixel-level image fusion: a survey of the state of the art. Inf. Fusion 83, 100–112 (2017).

  2. Xu, J., Yu, S., You, X., Leng, M., Jing, X.Y., Chen, C.: Multi-view hybrid embedding: a divide-and-conquer approach (2018)

  3. Yu, L., Tillo, T., Xiao, J., Grangetto, M.: Convolutional neural network for intermediate view enhancement in multiview streaming. IEEE Trans. Multimed. 20(1), 15–28 (2018)

    Article  Google Scholar 

  4. Gao, R., Vorobyov, S.A., Zhao, H.: Image fusion with cosparse analysis operator. IEEE Signal Process. Lett. 24(7), 943–947 (2017)

    Article  Google Scholar 

  5. Yang, B., Li, S.: Multi-focus image fusion and restoration with sparse representation. IEEE Trans. Instrum. Meas. 59(4), 884–892 (2010)

    Article  Google Scholar 

  6. Li, S., Yin, H., Fang, L.: Group-sparse representation with dictionary learning for medical image denoising and fusion. IEEE Trans. Biomed. Eng. 59(12), 3450–3459 (2012)

    Article  Google Scholar 

  7. Wang, J., Peng, J., Feng, X., He, G., Fan, J.: Fusion method for infrared and visible images by using non-negative sparse representation. Infr. Phys. Technol. 67, 477–489 (2014)

    Article  Google Scholar 

  8. Li, H., He, X., Tao, D., Tang, Y., Wang, R.: Joint medical image fusion, denoising and enhancement via discriminative low-rank sparse dictionaries learning. Pattern Recognit. 79, 130–146 (2018)

    Article  Google Scholar 

  9. Li, H., Manjunath, B.S., Mitra, S.K.: Multi-sensor image fusion using the wavelet transform. Graph Models Image Process 57(3), 235–245 (1995)

    Article  Google Scholar 

  10. Yang, Y., Tong, S., Huang, S., Lin. P.: Multi-focus image fusion based on NSCT and focused area detection. IEEE Sensors J. 15(5), 2824–2838, (2015).

  11. Yang, Y., Que, Y., Huang, S., Lin. P.: Multimodal sensor medical image fusion based on type-2 fuzzy logic in NSCT domain. IEEE Sensors J. 16(10), 3735–3745, (2016).

  12. Bhatnagar, G., Wu, Q.M.J., Liu, Z.: Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimed. 15(5), 1014–1024 (2013)

    Article  Google Scholar 

  13. Li, S., Kang, X., Hu, J.: Image fusion with guided filtering. IEEE Trans. Image Process. 22(7), 2864–2875 (2013)

    Article  Google Scholar 

  14. Liu, Y., Chen, X., Ward, R.K., Wang, Z.J.: Image fusion with convolutional sparse representation. IEEE Signal Process. Lett. 23(12), 1882–1886 (2016)

    Article  Google Scholar 

  15. Liu, Y., Chen, X., Ward, R.K., Wang, Z.J.: Medical image fusion via convolutional sparsity based morphological component analysis. IEEE Signal Process. Lett. 26(3), 485–489 (2019)

    Article  Google Scholar 

  16. Ma, J., Zhou, Z., Wang, B., Zong, H.: Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys. Technol. 82, 8–17 (2017)

    Article  Google Scholar 

  17. Li, H., Wu, J., Kittle, J.: MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Trans. Image Process. (2020). https://doi.org/10.1109/TIP.2020.2975984

    Article  Google Scholar 

  18. Liu, Y., Chen, X., Wang, Z., Wang, Z.J., Ward, R.K., Wang, X.: Deep learning for pixel-level image fusion: recent advances and future prospects. Inf. Fusion 42, 158–173 (2018)

    Article  Google Scholar 

  19. Liu, Y., Chen, X., Peng, H., Wang, Z.: Multi-focus image fusion with a deep convolutional neural network. Inf. Fusion 36, 191–207 (2017)

    Article  Google Scholar 

  20. Ma, J., Yu, W., Liang, P., Li, C., Jiang, J.: FusionGAN: a generative adversarial network for infrared and visible image fusion. Inf. Fusion 48, 11–26 (2019)

    Article  Google Scholar 

  21. Ma, J., Liang, P., Yu, W., Chen, C., Guo, X., Wu, J., Jiang, J.: Infrared and visible image fusion via detail preserving adversarial learning. Inf. Fusion 54, 85–98 (2020)

    Article  Google Scholar 

  22. Li, H., Wu, X.J.: DenseFuse: a fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2019)

    Article  MathSciNet  Google Scholar 

  23. Kumar, N., Sukavanam, N.: Weakly supervised deep network for spatiotemporal localization and detection of human actions in wild conditions. Vis. Comput. (2019). https://doi.org/10.1007/s00371-019-01777-5

    Article  Google Scholar 

  24. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  Google Scholar 

  25. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: NIPS (2015)

  26. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. Proc. IEEE Conf. Comput. Vis. Pattern Recog. 248–255 (2009).

  27. Li, H., Wu, X.-J.: Multi-focus image fusion using dictionary learning and low-rank representation. In: Zhao, Y., Kong, X., Taubman, D. (eds.) ICIG 2017. LNCS, vol. 10666, pp. 675–686. Springer, Cham (2017)

    Google Scholar 

  28. Fashandi, H., Peters, J., Ramanna, S.: L2 norm length based image similarity measures: concrescence of image feature histogram distances. In: Signal and Image Processing, International Association of Science Technology Development, pp. 178–185. Honolulu (2009)

  29. Zhang, X., Ma, Y., Fan, F., Zhang, Y., Huang, J.: Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition. J. Opt. Soc. Am. Opt. Image Sci. 34(8), 1400–1410, (2017).

  30. Toet, A.: TNO image fusion dataset. Figshare. Data (2014). https://figshare.com/articles/TN_Image_Fusion_Dataset/1008029.

  31. Xu, H., Ma, J., Guo, X., Ling, H.: U2Fusion: a unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2020.3012548

    Article  Google Scholar 

  32. Ma, J., Chen, C., Li, C., Huang, J.: Infrared and visible image fusion via gradient transfer and total variation minimization. Inf. Fusion 31, 100–109 (2016)

    Article  Google Scholar 

  33. Prabhakar, K.R., Srikar, V.S., Babu, R.V.: Deepfuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the IEEE international conference on computer vision, pp 4714–4722 (2017).

  34. Ma, J., Ma, Y., Li, C.: Infrared and visible image fusion methods and applications: a survey. Inf. Fusion 45, 153–178 (2019)

    Article  Google Scholar 

  35. Roberts, J.W., van Aardt, J., Ahmed, F.: Assessment of image fusion procedures using entropy, image quality, and multispectral classification. J. Appl. Remote Sens. 2, 023522 (2008)

    Article  Google Scholar 

  36. Qu, G., Zhang, D., Yan, P.: Information measure for performance of image fusion. Electron. Lett. 38(7), 313–315 (2002)

    Article  Google Scholar 

  37. Xydeas, C.S., Petrovic, V.: Objective image fusion performance measure. Electron. Lett. 36(4), 308–309 (2000)

    Article  Google Scholar 

  38. Yang, C., Zhang, J Q., Wang, X R., Liu. X.: A novel similarity based quality metric for image fusion. Inf. Fusion 9, 156–160, (2008).

  39. Eskicioglu, A.M., Fisher, P.S.: Image quality measures and their performance. IEEE Trans. Commun. 43, 2959–2965 (1995)

    Article  Google Scholar 

  40. Cui, G., Feng, H., Xu, Z., Li, Q., Chen, Y.: Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition. Opt. Commun. 341, 199–209 (2015)

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.61862030 and No.62072218), by the Natural Science Foundation of Jiangxi Province (No.20192ACB20002, and No.20192ACBL21008), and by the Talent project of Jiangxi Thousand Talents Program (No. jxsq2019201056).

Author information

Authors and Affiliations

  1. School of Information Technology, Jiangxi University of Finance and Economics, Nanchang, 330032, China

    Yong Yang, Jia-Xiang Liu, Hang-Yuan Lu & Wen-Ying Wen

  2. School of Computer Science and Technology, Tiangong University, Tianjin, 300387, China

    Shu-Ying Huang

Authors
  1. Yong Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jia-Xiang Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shu-Ying Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Hang-Yuan Lu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Wen-Ying Wen
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Shu-Ying Huang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Y., Liu, JX., Huang, SY. et al. VMDM-fusion: a saliency feature representation method for infrared and visible image fusion. SIViP 15, 1221–1229 (2021). https://doi.org/10.1007/s11760-021-01852-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-01852-2

Keywords