Abstract
In this paper, we propose a novel pseudo-end-to-end Pre-training multi-model image fusion network, termed CTFusion, to take advantage of convolution operations and vision transformer for multi-modal image fusion. Unlike existing pre-trained models that are based on public datasets, which contain two stages of training with a single input and a fusion strategy designed manually, our method is a simple single-stage pseudo-end-to-end model that uses a dual input adaptive fusion method and can be tested directly. Specifically, the fusion network first adopts a dual dense convolution network to obtain the abundant semantic information, and then the feature map is converted to a token and fed into a multi-path transformer fusion block to model the global-local information of sources images. Finally, we obtain the fusion image by a followed convolutional neural network block. Extensive experiments have been carried out on two publicly available multi-modal datasets, experiment results demonstrate that the proposed model outperforms state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, Y.K., Huang, W.B., Sun, F.C., Xu, T.Y., Rong, Y., Huang, J.: Deep multimodal fusion by channel exchanging. In: Advances in Neural Information Processing Systems, 33 (2020)
Li, H., Wu, X.J.: DenseFuse: a fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2018)
Naidu, V.: Hybrid DDCT-PCA based multi sensor image fusion. J. Opt. 43(1), 48–61 (2014)
Ram Prabhakar, K., Sai Srikar, V., Babu, R.: DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4714–4722 (2017)
Xu, H., Ma, J., Jiang, J., Guo, X., Ling, H.: U2fusion: a unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell. 44, 502–518 (2020)
Zhang, H., Xu, H., Xiao, Y., Guo, X., Ma, J.: Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 12797–12804 (2020)
Li, H., Wu, X.J., Kittler, J.: MDLatLRR: a novel decomposition method for infrared and visible image fusion. IEEE Trans. Image Process. 29, 4733–4746 (2020)
Ma, J., Xu, H., Jiang, J., Mei, X., Zhang, X.P.: DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 29, 4980–4995 (2020)
Zhang, H., Le, Z., Shao, Z., Xu, H., Ma, J.: MFF-GAN: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Inf. Fusion 66, 40–53 (2021)
Hu, J., Shen, L., Sun, G., Albanie, S.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. 99 (2017)
Wang, X.L., Ross, G., Abhinav, G., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Ashish, V., et al.: Attention is all you need. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021)
Xie, E., et al.: SegFormer: simple and efficient design for semantic segmentation with transformers, in arXiv preprint arXiv:2105.15203 (2021)
Prakash, A., Chitta, K., Geiger, A.: Multi-modal fusion transformer for end-to-end autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7077–7087 (2021)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Wu, H.P., et al.: CVT: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808 (2021)
Xiao, T., Dollar, P., Singh, M., Mintun, E., Darrell, T., Girshick, R.: Early convolutions help transformers see better. In: Thirty-Fifth Conference on Neural Information Processing Systems (2021)
Islam, Md.A., Jia, S., Bruce, N.D.B.: How much position information do convolutional neural networks encode? In: International Conference on Learning Representations (2020)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Zhou, Z., Dong, M., Xie, X., Gao, Z.: Fusion of infrared and visible images for night-vision context enhancement. Appl. Opt. 55(23), 6480–6490 (2016)
Mohammad, H., Masoud Amirkabiri, R.: Fast-FMI: non-reference image fusion metric. In: 2014 IEEE 8th International Conference on Application of Information and Communication Technologies (AICT), pp. 1–3 (2014)
Aslantas, V., Bendes, E.: A new image quality metric for image fusion: the sum of the correlations of differences. AEU-Int. J. Electron. Commun. 69(12), 1890–1896 (2015)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Shreyamsha Kumar, B.K.: Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform. SIViP 7(6), 1125–1143 (2012). https://doi.org/10.1007/s11760-012-0361-x
Han, S.S., Li, H.T., Gu, H.Y., et al.: The study on image fusion for high spatial resolution remote sensing images. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XXXVII Part B 7, 1159–1164 (2008)
Kumar, B.S.: Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform. Signal Image Video Process. 7(6), 1125–1143 (2013)
Zhang, Y., Liu, Y., Sun, P., Yan, H., Zhao, X., Zhang, L.: IFCNN: a general image fusion framework based on convolutional neural network. Inf. Fusion 54, 99–118 (2020)
Li, W., Xie, Y., Zhou, H., Han, Y., Zhan, K.: Structure-aware image fusion. Optik 172, 1–11 (2018)
Ma, K., Zeng, K., Wang, Z.: Perceptual quality assessment for multi-exposure image fusion. IEEE Trans. Image Process. 24(11), 3345–3356 (2015)
Acknowledgments
This work was partially supported by the Scientific Innovation 2030 Major Project for New Generation of AI under Grant 2020AAA0107300.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shen, Z., Wang, J., Pan, Z., Wang, J., Li, Y. (2022). CTFusion: Convolutions Integrate with Transformers for Multi-modal Image Fusion. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13534. Springer, Cham. https://doi.org/10.1007/978-3-031-18907-4_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-18907-4_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18906-7
Online ISBN: 978-3-031-18907-4
eBook Packages: Computer ScienceComputer Science (R0)