Abstract
As a powerful and continuously sought-after medical assistance technique, multimodal medical image fusion integrates the useful information from different single-modal medical images into a fused one. Nevertheless, existing deep learning-based methods often feed source images into a single network without considering the information among different channels and scales, which may inevitably lose the important information. To solve this problem, we proposed a multimodal medical image fusion method based on multichannel aggregated network. By iterating different residual densely connected blocks to efficiently extract the image features at three scales, as well as extracting the spatial domain, channel and fine-grained feature information of the source image at each scale separately. Simultaneously, we introduced multispectral channel attention to address the global average pooling problem of the vanilla channel attention mechanism. Extensive fusion experiments demonstrated that the proposed method surpasses some representative state-of-the-art methods in terms of both subjective and objective evaluation. The code of this work is available at https://github.com/JasonWong30/MCAFusion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Li, X., Guo, X., Han, P., Wang, X., Li, H., Luo, T.: Laplacian redecomposition for multimodal medical image fusion. IEEE Trans. Instrum. Meas.Instrum. Meas. 69(9), 6880–6890 (2020)
Li, X., Zhou, F., Tan, H.: Joint image fusion and denoising via three-layer decomposition and sparse representation. Knowl.-Based Syst..-Based Syst. 224, 107087 (2021)
Jie, Y., Zhou, F., Tan, H., Wang, G., Cheng, X., Li, X.: Tri-modal medical image fusion based on adaptive energy choosing scheme and sparse representation. Measurement 204(30), 112038 (2022)
Li, X., Wan, W., Zhou, F., Cheng, X., Jie, Y., Tan, H.: Medical image fusion based on sparse representation and neighbor energy activity. Biomed. Sig. Process. Control 80, 104353 (2023)
Zhu, Z., He, X., Qi, G., Li, Y., Cong, B., Liu, Y.: Brain tumor segmentation based on the fusion of deep semantics and edge information in multimodal MRI. Inf. Fusion 91, 376–387 (2023)
Li, X., Zhou, F., Tan, H., Zhang, W., Zhao, C.: Multimodal medical image fusion based on joint bilateral filter and local gradient energy. Inf. Sci. 569, 302–325 (2021)
Ma, J., Xu, H., Jiang, J., Mei, X., Zhang, X.-P.: DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 29, 4980–4995 (2020)
Xu, H., Ma, J., Le, Z., Jiang, J., Guo, X.: FusionDN: a unified densely connected network for image fusion. In: Proceedings of the AAAI Conference on Artificial Intelligence, New York, pp. 12484–12491 (2020)
Xu, H., Ma, J., Jiang, J., Guo, X., Ling, H.: U2Fusion: a unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell.Intell. 44(1), 502–518 (2020)
Fu, J., Li, W., Du, J., Huang, Y.: A multiscale residual pyramid attention network for medical image fusion. Biomed. Sig. Process. Control 66, 102488 (2021)
Zhang, H., Ma, J.: SDNet: a versatile squeeze-and-decomposition network for real-time image fusion. Int. J. Comput. VisionComput. Vision 129, 2761–2785 (2021)
Tang, W., He, F., Liu, Y., Duan, Y.: MATR: multimodal medical image fusion via multiscale adaptive transformer. IEEE Trans. Image Process. 31, 5134–5149 (2022)
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 43, 2480–2495 (2021)
Tang, L., Yuan, J., Ma, J.: Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network. Inf. Fusion 82, 28–42 (2022)
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057. PMLR, Lille (2015)
Chen, C., Gong, D., Wang, H., Li, Z., Wong, K.-Y.K.: Learning spatial attention for face super-resolution. IEEE Trans. Image Process. 30, 1219–1231 (2020)
Qin, Z., Zhang, P., Wu, F., Li, X.: FCAnet: frequency channel attention networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, pp. 783–792 (2021)
Tan, W., Tiwari, P., Pandey, H.M., Moreira, C., Jaiswal, A.K.: Multimodal medical image fusion algorithm in the era of big data. Neural Comput. Appl. 1–21 (2020)
Wang, Q., Shen, Y., Zhang, J.Q.: A nonlinear correlation measure for multivariable data set. Physica D D 200(3–4), 287–295 (2005)
Chen, Y., Blum, R.S.: A new automated quality assessment algorithm for image fusion. Image Vis. Comput.Comput. 27(10), 1421–1432 (2009)
Han, Y., Cai, Y., Cao, Y., Xu, X.: A new image fusion performance metric based on visual information fidelity. Inf. Fusion 14(2), 127–135 (2013)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Eskicioglu, A.M., Fisher, P.S.: Image quality measures and their performance. IEEE Trans. Commun.Commun. 43(12), 2959–2965 (1995)
Liu, Z., Blasch, E., Xue, Z., Zhao, J., Laganiere, R., Wu, W.: Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Trans. Pattern Anal. Mach. Intell.Intell. 34(1), 94–109 (2012)
Aslantas, V., Bendes, E.: A new image quality metric for image fusion: the sum of the correlations of differences. AEU-Int. J. Electron. Commun. 69(12), 1890–1896 (2015)
Acknowledgements
This research was supported by the National Natural Science Foundation of China (Grant Nos. 62201149, 62271148, 62201151).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, J., Li, X., Tan, H., Cheng, X. (2023). Multimodal Medical Image Fusion Based on Multichannel Aggregated Network. In: Lu, H., et al. Image and Graphics . ICIG 2023. Lecture Notes in Computer Science, vol 14359. Springer, Cham. https://doi.org/10.1007/978-3-031-46317-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-46317-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46316-7
Online ISBN: 978-3-031-46317-4
eBook Packages: Computer ScienceComputer Science (R0)