Abstract
The existing methodologies for image fusion in deep learning predominantly focus on convolutional neural networks (CNN). However, recent investigations have explored the utilization of Transformer models to enhance fusion performance. Hence, we propose a novel generative adversarial fusion framework that combines both CNN and Transformer, referred to as the Cross-Domain Bidirectional Interaction Fusion Network (CDBIFusion). Specifically, we devise three distinct pathways for the generator, each serving a unique purpose. The two CNN pathways are employed to capture local information from MRI and PET images, and the other pathway adopts a transformer architecture that cascades both source images as input, enabling the exploitation of global correlations. Moreover, we present a cross-domain bidirectional interaction (CDBI) module that facilitates the retention and interaction of information deactivated by the ReLU activation function between two CNN paths. The interaction operates by cross-cascading ReLU activation features and deactivation features from separate paths by two ReLU rectifiers and then delivering them to the other path, thus reducing the valuable information lost through deactivation. Extensive experiments have demonstrated that our CDBIFusion surpasses other current methods in terms of subjective perception and objective evaluation.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Zhao, Z., et al.: Cddfuse: correlation-driven dual-branch feature decomposition for multi-modality image fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5906–5916 (2023)
Zhao, Z., et al.: Equivariant Multi-Modality Image Fusion. ArXiv abs/2305.11443 (2023)
Al-Mualla, M.: Ebrahim, bull, david, hill, paul: perceptual image fusion using wavelets. IEEE Trans. Image Process. 26, 1076–1088 (2017)
Du, J., Li, W., Xiao, B., Nawaz, Q.: Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing 194, 326–339 (2016)
Singh, S., Gupta, D., Anand, R.S., Kumar, V.: Nonsubsampled shearlet based CT and MR medical image fusion using biologically inspired spiking neural network. Biomed. Signal Process. Control (2015)
Bhatnagar, G., Wu, Q., Zheng, L.: Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimedia 9, 1014–1024 (2014)
Liu, Y., Wang, Z.: Simultaneous image fusion and denoising with adaptive sparse representation. Image Process. Iet 9, 347–357 (2014)
Yu, L., Xun, C., Cheng, J., Hu, P.: A medical image fusion method based on convolutional neural networks. In: 2017 20th International Conference on Information Fusion (2017)
Wang, K., Zheng, M., Wei, H., Qi, G., Li, Y.: Multi-modality medical image fusion using convolutional neural network and contrast pyramid. Sensors 20, 2169 (2020)
Lahoud, F., Süsstrunk, S.: Zero-learning fast medical image fusion. In: 2019 22th International Conference on Information Fusion (2019)
Song, S., Wang, J., Wang, Z., Su, J., Ding, X., Dang, K.: Bilateral-Fuser: A Novel Multi-cue Fusion Architecture with Anatomical-aware Tokens for Fovea Localization. arXiv preprint arXiv:2302.06961 (2023)
Ma, J., Yu, W., Liang, P., Li, C., Jiang, J.: FusionGAN: a generative adversarial network for infrared and visible image fusion. Inform. Fus. 48, 11–26 (2019)
Ma, J., Xu, H., Jiang, J., Mei, X., Zhang, X.P.: DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. (2020)
Guo, X., Nie, R., Cao, J., Zhou, D., Mei, L., He, K.: FuseGAN: learning to fuse multi-focus image via conditional generative adversarial network. IEEE Trans. Multimedia 1982–1996 (2019)
Ma, J., Zhang, H., Shao, Z., Liang, P., Xu, H.: GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion. IEEE Trans. Instrument. Measure. 1 (2020)
Hao, Z.A., Zl, A., Zs, B., Han, X.A., Jm, A.: MFF-GAN: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Inform. Fusion 66, 40–53 (2021)
Wang, Z., Shao, W., Chen, Y., Xu, J., Zhang, X.: Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning (2022)
Ma, T., Li, B., Liu, W., Hua, M., Dong, J., Tan, T.: CFFT-GAN: cross-domain Feature Fusion Transformer for Exemplar-based Image Translation. ArXiv abs/2302.01608, (2023)
Hu, Q., Guo, X.: Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation (2021)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022 (2021)
Kumar, B.S.: Image fusion based on pixel significance using cross bilateral filter. Signal, image and video processing (2015)
Dhuli, Ravindra, Bavirisetti, Prasad, D.: Two-scale image fusion of visible and infrared images using saliency detection. Infrared Phys. Technol. (2016)
Bavirisetti, D.P.: Multi-sensor image fusion based on fourth order partial differential equations. In: 20th International Conference on Information Fusion (2017)
Li, H., Wu, X.J.: Infrared and visible image fusion using Latent Low-Rank Representation (2018)
Xu, H., Ma, J., Jiang, J., Guo, X., Ling, H.: U2Fusion: a unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell. 1–1 (2020)
Xu, H., Ma, J.: EMFusion: an unsupervised enhanced medical image fusion network. Inform. Fusion 76, 177–186 (2021)
Tang, L., Yuan, J., Ma, J.: Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network. Inform. Fusion 82, 28–42 (2022)
Xu, H., Gong, M., Tian, X., Huang, J., Ma, J.: CUFD: an encoder–decoder network for visible and infrared image fusion based on common and unique feature decomposition. Comput. Vis. Image Understand. 218 (2022)
Li, W., Peng, X., Fu, J., Wang, G., Huang, Y., Chao, F.: A multiscale double-branch residual attention network for anatomical–functional medical image fusion. Comput. Biol. Med. 141, 105005 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, J., Li, B., Wang, B., Shao, Z., Huang, J., Lu, J. (2024). CDBIFusion: A Cross-Domain Bidirectional Interaction Fusion Network for PET and MRI Images. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14437. Springer, Singapore. https://doi.org/10.1007/978-981-99-8558-6_36
Download citation
DOI: https://doi.org/10.1007/978-981-99-8558-6_36
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8557-9
Online ISBN: 978-981-99-8558-6
eBook Packages: Computer ScienceComputer Science (R0)