Abstract
The aim of medical image fusion is to integrate complementary information in multi-modality medical images into an informative fused image which is pivotal for assistance in clinical diagnosis. Since medical images in different modalities always have great variety of characteristics (such as resolution and functional information), the fused images obtained from traditional methods would be blurred in details or loss of information in some degree. To solve these problems, we propose a novel deep learning-based multi-resolution medical image fusion network with iterative back-projection (IBPNet) in this paper. In our IBPNet, up-projection and down-projection blocks are designed to achieve feature maps alternation between high- and low-resolution images. The feedback errors generated in the alternation process are self-corrected in the reconstruction process. Moreover, an effective combined loss function is designed, which can adapt to multi-resolution medical image fusion task. High spatial resolution magnetic resonance imaging (MRI) images and low spatial resolution color Positron emission tomography (PET) images are used to demonstrate the validation of the proposed method. Experimental results show that our method is superior to other state-of-the-art fusion methods in terms of visual perception and objective evaluation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Yousif, A.S., Omar, Z.B., Sheikh, U.U.: A survey on multi-scale medical images fusion techniques: brain diseases. J. Biomed. Eng. Med. Imaging 7(1), 18–38 (2020)
Wei, T., Tiwari, P., Pandey, H.M., Moreira, C., Jaiswal, A.K.: Multimodal medical image fusion algorithm in the era of big data. Neural Comput. Appl. 3, 1–21 (2020)
Das, S., Kundu, M.K.: A neuro-fuzzy approach for medical image fusion. IEEE Trans. Biomed. Eng. 60(12), 3347–3353 (2013)
Yin, M., Liu, X., Liu, Y., Chen, X.: Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 68(1), 49–64 (2018)
Liu, Y., Wang, Z.: Simultaneous image fusion and denoising with adaptive sparse representation. IET Image Proc. 9(5), 347–357 (2014)
Liu, Y., Chen, X., Ward, K.R., Wang, Z.J.: Medical image fusion via convolutional sparsity based morphological component analysis. IEEE Signal Process. Lett. 26(3), 485–489 (2019)
Huang, J., Le, Z., Ma, Y., Fan, F., Yang, L.: MGMDcGAN: medical image fusion using multi-generator multi-discriminator conditional generative adversarial network. IEEE Access 8, 55145–55157 (2020)
Lahoud, F., Sabine S.: Zero-learning fast medical image fusion. In: The Proceedings of 22th International Conference on Information Fusion, Ottawa, ON, Canada, pp. 1–8. IEEE (2019)
Ma, J., Xu, H., Jiang, J., Mei, X., Zhang, X.P.: DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 29, 4980–4995 (2020)
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: Proceedings of International Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA (2018)
Irani, M., Peleg, S.: Improving resolution by image registration. GVGIP: Graph. Models Image Process. 53(3), 231–239 (1991)
Dai, S., Mei, H., Ying, W., Gong, Y.: Bilateral back-projection for single image super resolution. In the Proceedings of International Conference on Multimedia and Expo, Beijing, China, pp. 1039–1042. IEEE (2008)
Dong, W., Zhang, L., Shi, G., Wu, X.: Nonlocal back-projection for adaptive image enlargement. In: Proceedings of International Conference on Image Processing, Piscataway, NJ, pp. 348–352. IEEE (2010)
Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of CVF Conference on Computer Vision and Pattern Recognition, UT, Salt Lake City, pp. 1664–1673. IEEE (2018)
Timofte, R., Rothe, R., Gool, L.V.: Seven ways to improve example-based single image super resolution. In the Proceedings of International Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 1865–1873. IEEE (2016)
Zhang, H., Xu, H., Xiao, Y., Guo, X., Ma, J.: Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity. In the Proceedings of 34th AAAI Conference on Artificial Intelligence, New York, USA, pp. 12797–12804 (2020)
Xydeas, C.S., Petrovic, V.: Objective image fusion performance measure. Electron. Lett. 36(4), 308–309 (2000)
Piella, G., Heijmans, H.: A new quality metric for image fusion. In: The Proceedings of International Conference on Image Processing, pp. 173–176. Barcelona, Spain. IEEE (2003)
Zhou, W., Bovik, A.C.: A universal image quality index. IEEE Signal Process. Lett. 9(3), 81–84 (2002)
Acknowledgments
This paper is supported by the National Natural Science Foundation of China (Nos.61871210), Chuanshan Talent Project of the University of South China.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, C., Yang, B. (2021). A Multi-resolution Medical Image Fusion Network with Iterative Back-Projection. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13021. Springer, Cham. https://doi.org/10.1007/978-3-030-88010-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-88010-1_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88009-5
Online ISBN: 978-3-030-88010-1
eBook Packages: Computer ScienceComputer Science (R0)