Abstract
The vision sensor is capable of capturing image detail features suitable for human observation, while the infrared sensor is capable of capturing the thermal characteristics of the target object. Therefore, the vision and infrared image fusion aim to retain both the rich detail features in the visual image and the thermal characteristics of the infrared image. This study proposes a novel deep U-Net network model to solve the fusion task. First, an improved deep model is proposed for better feature extraction by borrowing the process of decomposition, fusion and reconstruction in the multiscale decomposition process. Second, structural similarity is introduced into the loss function, which enables the network to enhance the quality of the detailed features of the generated images. Third, we propose a new hierarchical fusion strategy as well as average fusion and weighted fusion rules. Extensive experiments demonstrate that the proposed algorithm is superior to state-of-the-art algorithms.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Babenko A, Slesarev A, Chigorin A, Lempitsky V (2014) Neural codes for image retrieval. In: European conference on computer vision. Springer, Cham, pp 584–599
Benediktsson A, Ghamisi P, Mura Mauro D (2015) A survey on spectral-spatial classification techniques based on attribute profiles. IEEE Trans Geosci Remote Sens 3(53):2335–2353
Chavez P, Sides S, Anderson J (1991) Comparison of three different methods to merge multiresolution and multispectral data-Landsat TM and SPOT panchromatic. Photogramm Eng Remote Sens 3(57):295–303
Cheng M, Wang C, Li J (2014) Sparse representation based pansharpening using trained dictionary. IEEE Geosci Remote Sens Lett 1(11):293–297
Cunha A, Zhou J, Do M (2006) The nonsubsampled contourlet transform: theory, design, and applications. IEEE Trans Image Process 10(15):3089–3101
Ghassemian H (2016) A review of remote sensing image fusion methods. Inf Fusion 1(32):75–89
Girshick R (2015) Fast R-CNN. In: 2015 IEEE international conference on computer vision (ICCV)
Guo M, Zhang H, Li J, Zhang L, Shen H (2014) An Online coupled dictionary learning approach for remote sensing image fusion. IEEE J Sel Top Appl Earth Observ Remote Sens 4(7):1284–1294
Huang G, Liu Z, Maaten L, Weinberger K (2017) Densely connected convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2261–2269
Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
Lecun Y, Boser B, Denker J, Henderson D, Howard R, Hubbard W, Jackel L (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 4(1):541–551
Lecun L, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 11(86):2278–2324
Lewis JJ, O’Callaghan RJ, Nikolov SG, Bull DR, Canagarajah N (2007) Pixel-and region-based image fusion with complex wavelets. Inf Fusion 8(2):119–130
Li H, Wu XJ (2018) DenseFuse: a fusion approach to infrared and visible images. IEEE Trans Image Process 28(5):2614–2623
Li Y, Sun Y, Huang X, Qi G, Zheng M, Zhu Z (2018) An image fusion method based on sparse representation and sum modified-Laplacian in NSCT domain. Entropy 20(7):522
Liu X, Mei W, Du H (2018) Multi-modality medical image fusion based on image decomposition framework and nonsubsampled shearlet transform. Biomed Signal Process Control 40:343–350
Liu X, Liu Q, Wang Y (2020) Remote sensing image fusion based on two-stream fusion network. Inf Fusion 55:1–15
Ma J, Yu W, Chen C, Liang P, Jiang J (2020) Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Inf Fusion 62:110–120
Moonon A, Hu J, Li S (2015) Remote sensing image fusion method based on nonsubsampled shearlet transform and sparse representation. Sens Imag 1(16):23
Pan Y, Pi D, Chen J, Meng H (2021a) FDPPGAN: remote sensing image fusion based on deep perceptual patchGAN. Neural Comput Appl 33(15):9589–9605
Pan Y, Pi D, Khan I, Khan Z, Meng H (2021b) Densenetfuse: a study of deep unsupervised densenet to infrared and visual image fusion. J Ambient Intell Human Comput 11:10339–10351
Prabhakar RK, Sai Srikar V, Venkatesh Babu R (2017) DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the IEEE international conference on computer vision, pp 4714–4722
Qin X, Ji C, Shen Y, Wang P, Zhang J (2021) ECT image recognition of pipe plugging flow patterns based on broad learning system in mining filling. Adv Civil Eng 20:1–7
Ranchin T, Wald L (2000) Fusion of high spatial and spectral resolution images: the arsis concept and its implementation. Photogramm Eng Remote Sens 1(66):49–61
Starck J, Candes E, Donoho D (2002) The curvelet transform for image denoising. IEEE Trans Image Process 6(11):670–684
Tu T (2005) Adjustable intensity-hue-saturation and Brovey transform fusion technique for IKONOS/QuickBird imagery. Opt Eng 11(44):116201
Tu T, Su S, Shyu H, Huang P (2001) A new look at IHS-like image fusion methods. Inf Fusion 2(3):177–186
Tu T, Huang P, Hung C, Chang C (2004) A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geosci Remote Sens Lett 4(1):309–312
Wang W, Jiao L, Yang S (2014) Fusion of multispectral and panchromatic images via sparse representation and local autoregressive model. Inf Fusion 20:73–87
Wang L, Liu X, Chen D, Yang H, Wang C (2020) ECT image reconstruction algorithm based on multiscale dual-channel convolutional neural network. Complexity 4918058
Xu J, Yu X, Pei W, Hu D, Zhang L (2015) A remote sensing image fusion method based on feedback sparse component analysis. Comput Geosci 2(85):115–123
Xu S, Zhang J, Zhao Z, Sun K, Zhang C (2021) Deep gradient projection networks for pan-sharpening. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)
Yang Y, Ren W, Hu X, Li K, Cao X (2021) SRGAT: single image super-resolution with graph attention network. IEEE Trans Image Process 30:4905–4918
Yin M, Liu X, Liu Y, Chen X (2018) Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans Instrum Meas 68(1):49–64
Yuan Q, Wei Y, Meng X, Shen H, Zhang L (2018) A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE J Sel Top Appl Earth Observ Remote Sens 3(11):978–989
Zheng S, Shi W, Liu J, Tian J (2008) Remote sensing image fusion using multiscale mapped LS-SVM. IEEE Trans Geosci Remote Sens 5(46):1313–1322
Zhou W, Bovik A (2002) A universal image quality index. IEEE Signal Process Lett 3(9):81–84
Zhu X, Bamler R (2013) A sparse image fusion algorithm with application to pan-sharpening. IEEE Trans Geosci Remote Sens 5(51):2827–2836
Acknowledgements
This work is supported by National Science and Technology Innovation 2030-Key Project of "New Generation Artificial Intelligence" under Grant 2021ZD0113103, the Natural Science Foundation of Jiangsu Provincial Higher Education under Grant 19KJB520008, and in part by the Young Scholar Support Program of Nanjing University of Finance and Economics, and in part by the Educational Reform Project of Nanjing University of Finance and Economics under Grant JGY19060.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Pan, Y., Pi, D., Khan, I.A. et al. DUFuse: Deep U-Net for visual and infrared images fusion. J Ambient Intell Human Comput 14, 12549–12561 (2023). https://doi.org/10.1007/s12652-022-04323-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-022-04323-9