Skip to main content
SpringerLink
Log in
Book cover

International Conference on Image and Graphics

ICIG 2021: Image and Graphics pp 40–51Cite as

MSC-Fuse: An Unsupervised Multi-scale Convolutional Fusion Framework for Infrared and Visible Image

  • Conference paper
  • First Online:

  • 2024 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12888))

Abstract

Lacking the labeled data, how to establish an unsupervised learning method is essential for the infrared and visible image fusion task. As such, this article introduces a novel unsupervised learning fusion framework. Our proposed framework consists of three components: encoder, fusion layer, and decoder, respectively. Firstly, an encoder is designed to extract salient features from multiple source images. With the multi-scale convolution modules, the encoder can produce more useful features. Then these features are fused at the fusion layer. Finally, the decoder reconstructs the fused features to generate the fused image. To achieve the unsupervised training of the network, a no-reference quality metric and a pixel-level function are utilized to calculate the loss function. Experimental results show that compared with other fusion methods, our proposed method can achieve better performance in both objective and subjective assessments.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Li, S., Kang, X., Fang, L., Hu, J., Yin, H.: Pixel-level image fusion: a survey of the state of the art. Inf. Fusion 33, 100–112 (2017)

    Article  Google Scholar 

  2. Ma, J., Ma, Y., Li, C.: Infrared and visible image fusion methods and applications: a survey. Inf. Fusion 33, 153–178 (2019)

    Article  Google Scholar 

  3. Ben, H.A., Yun, H., Hamid, K., Alan, W.: A multiscale approach to pixel-level image fusion. Integr. Comput. Aided Eng. 12(2), 135–146 (2005)

    Article  Google Scholar 

  4. Bulanon, D.M., Burks, T.F., Alchanatis, V.: Image fusion of visible and thermal images for fruit detection. Biosys. Eng. 103(1), 12–22 (2009)

    Article  Google Scholar 

  5. Ming, Z., Ji-Gang, S., Wei, L., Li-Qiang, G.: Multiple multifocus color image fusion using quaternion curvelet transform. Opt. Precis. Eng. 21(10), 2671–2678 (2013)

    Article  Google Scholar 

  6. Yang, B., Li, S.: Multifocus image fusion and restoration with sparse representation. IEEE Trans. Instrum. Meas. 59(4), 884–892 (2010)

    Article  Google Scholar 

  7. Zhang, Q., Fu, Y., Li, H., Zou, J.: Dictionary learning method for joint sparse representation-based image fusion. Opt. Eng. 52(5), 7006 (2013)

    Google Scholar 

  8. Li, H., Wu, X.-J.: Multi-focus image fusion using dictionary learning and low-rank representation. In: Zhao, Y., Kong, X., Taubman, D. (eds.) ICIG 2017. LNCS, vol. 10666, pp. 675–686. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71607-7_59

    Chapter  Google Scholar 

  9. Li, B., Wu, W., Wang, Q., Zhang, F., Yan, J.: Siamrpn++: evolution of siamese visual tracking with very deep networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  10. Peng, S., Jiang, W., Pi, H., Li, X., Bao, H., Zhou, X.: Deep snake for real-time instance segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  11. Yang, C., Xu, Y., Shi, J., Dai, B., Zhou, B.: Temporal pyramid network for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  12. Liu, Y., Chen, X., Peng, H., Wang, Z.: Multi-focus image fusion with a deep convolutional neural network. Inf. Fusion 36, 191–207 (2017)

    Article  Google Scholar 

  13. Li, H., Wu, X.J., Kittler, J.: Infrared and Visible Image Fusion using a Deep Learning Framework. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2705–2710. IEEE (2018)

    Google Scholar 

  14. Li, H., Wu, X.J.: Densefuse: a fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2019)

    Article  MathSciNet  Google Scholar 

  15. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Zitnick, C.L.: Microsoft coco: common objects in context (2014)

    Google Scholar 

  16. Song, X., Wu, X.-J., Li, H.: MSDNet for medical image fusion. In: Zhao, Y., Barnes, N., Chen, B., Westermann, R., Kong, X., Lin, C. (eds.) ICIG 2019. LNCS, vol. 11902, pp. 278–288. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34110-7_24

    Chapter  Google Scholar 

  17. Ram Prabhakar, K., Sai Srikar, V., Venkatesh Babu, R.: Deepfuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4714–4722 (2017)

    Google Scholar 

  18. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)

    Google Scholar 

  19. Wang, Z.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  20. Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2017)

    Article  Google Scholar 

  21. Ma, K., Zeng, K., Wang, Z.: Perceptual quality assessment for multi-exposure image fusion. IEEE Trans. Image Process. 24(11), 3345 (2015)

    Article  MathSciNet  Google Scholar 

  22. Toet, A.: TNO Image Fusion Dataset (2014)

    Google Scholar 

  23. Ma, J., Zhou, Z., Wang, B., Zong, H.: Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys. Technol. 82, 8–17 (2017)

    Article  Google Scholar 

  24. Roberts, J.W., Van Aardt, J.A., Ahmed, F.B.: Assessment of image fusion procedures using entropy, image quality, and multispectral classification. J. Appl. Remote Sens. 2(1), 023522 (2008)

    Google Scholar 

  25. Rao, Y.J.: In-fibre bragg grating sensors. Meas. Sci. Technol. 8(4), 355 (1997)

    Article  Google Scholar 

  26. Li, S., Yang, B.: Multifocus image fusion using region segmentation and spatial frequency. Image Vis. Comput. 26(7), 971–979 (2008)

    Article  Google Scholar 

  27. Han, Y., Cai, Y., Cao, Y., Xu, X.: A new image fusion performance metric based on visual information fidelity. Inf. Fusion 14(2), 127–135 (2013)

    Article  Google Scholar 

  28. Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 8, 1226–1238 (2005)

    Article  Google Scholar 

  29. Xydeas, C.S., Pv, V.: Objective image fusion performance measure. Mil. Tech. Cour. 56(4), 181–193 (2000)

    Google Scholar 

  30. Shreyamsha Kumar, B.K.: Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform. SIViP 7(6), 1125–1143 (2012). https://doi.org/10.1007/s11760-012-0361-x

    Article  Google Scholar 

  31. Liu, C.H., Qi, Y., Ding, W.: Infrared and visible image fusion method based on saliency detection in sparse domain. Infrared Phys. Technol. 83, 94–102 (2017)

    Article  Google Scholar 

  32. Ma, J., Chen, C., Li, C., Huang, J.: Infrared and visible image fusion via gradient transfer and total variation minimization. Inf. Fusion 31, 100–109 (2016)

    Article  Google Scholar 

  33. Liu, Y., Chen, X., Ward, R.K., Wang, Z.J.: Image fusion with convolutional sparse representation. IEEE Signal Process. Lett. 23(12), 1882–1886 (2016)

    Article  Google Scholar 

  34. Ma, J., Yu, W., Liang, P., Li, C., Jiang, J.: FusionGAN: a generative adversarial network for infrared and visible image fusion. Inf. Fusion 48, 11–26 (2019)

    Article  Google Scholar 

  35. Zhang, Y., Liu, Y., Sun, P., Yan, H., Zhao, X., Zhang, L.: IFCNN: a general image fusion framework based on convolutional neural network. Inf. Fusion 54, 99–118 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi, 214122, Jiangsu, China

    Guo-Yang Chen, Xiao-Jun Wu, Hui Li & Tian-Yang Xu

Authors
  1. Guo-Yang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiao-Jun Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tian-Yang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Yuxin Peng

  2. Tsinghua University, Beijing, China

    Shi-Min Hu

  3. Tampere University, Tampere, Finland

    Moncef Gabbouj

  4. Zhejiang University, Hangzhou, China

    Kun Zhou

  5. Technion – Israel Institute of Technology, Haifa, Israel

    Michael Elad

  6. Tsinghua University, Beijing, China

    Kun Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, GY., Wu, XJ., Li, H., Xu, TY. (2021). MSC-Fuse: An Unsupervised Multi-scale Convolutional Fusion Framework for Infrared and Visible Image. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12888. Springer, Cham. https://doi.org/10.1007/978-3-030-87355-4_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87355-4_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87354-7

  • Online ISBN: 978-3-030-87355-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation