skip to main content
10.1145/3603781.3603940acmotherconferencesArticle/Chapter ViewAbstractPublication PagescniotConference Proceedingsconference-collections
research-article

Multi-scale Feature Fusion Convolutional Neural Network for Multi-Modal Medical Image Fusion

Published: 27 July 2023 Publication History

Abstract

Compared with images in general scenes, multi-modal medical images contain more detailed features and require higher integrity of features. Therefore, when fusing multi-modal medical images, features at different scales need to be accurately extracted to ensure the above requirements, which cannot be done in general convolutional neural network (CNN). To solve this problem, a convolutional neural network based on multi-scale feature fusion is proposed to improve the fusion quality of multi-modal medical image. Specifically, the proposed network consists of two trunks and three branches to extract features at different scales. The trunks and branches are connected by the fusion modules (FM) to realize the fusion of multi-scale features. Finally, the fused multi-scale features are extracted by multiple convolutions and concatenated with the features of the trunks to reconstruct and generate the fused image. The results of the objective and subjective evaluation show that the proposed method is advanced in most of the indexes compared with other state-of-the-art methods.

References

[1]
M Azam, K Khan, Muhammad Ahmad, and Manuel Mazzara. 2021. Multimodal medical image registration and fusion for quality enhancement. Cmc-Comput 68 (2021), 821–840.
[2]
Muhammad Adeel Azam, Khan Bahadar Khan, Sana Salahuddin, Eid Rehman, Sajid Ali Khan, Muhammad Attique Khan, Seifedine Kadry, and Amir H Gandomi. 2022. A review on multimodal medical image fusion: Compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics. Computers in biology and medicine 144 (2022), 105253.
[3]
Satishkumar S Chavan, Abhishek Mahajan, Sanjay N Talbar, Subhash Desai, Meenakshi Thakur, and Anil D’cruz. 2017. Nonsubsampled rotated complex wavelet transform (NSRCxWT) for medical image fusion related to clinical aspects in neurocysticercosis. Computers in biology and medicine 81 (2017), 64–78.
[4]
Jiao Du, Weisheng Li, Ke Lu, and Bin Xiao. 2016. An overview of multi-modal medical image fusion. Neurocomputing 215 (2016), 3–20.
[5]
Ruichao Hou, Dongming Zhou, Rencan Nie, Dong Liu, and Xiaoli Ruan. 2019. Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model. Medical & biological engineering & computing 57 (2019), 887–900.
[6]
Weiwei Kong, Qiguang Miao, and Yang Lei. 2018. Multimodal sensor medical image fusion based on local difference in non-subsampled domain. IEEE Transactions on Instrumentation and Measurement 68, 4 (2018), 938–951.
[7]
Baiying Lei, Siping Chen, Dong Ni, and Tianfu Wang. 2016. Discriminative learning for Alzheimer’s disease diagnosis via canonical correlation analysis and multimodal fusion. Frontiers in aging neuroscience 8 (2016), 77.
[8]
Shuaiqi Liu, Mingzhu Shi, Zhihui Zhu, and Jie Zhao. 2017. Image fusion based on complex-shearlet domain with guided filtering. Multidimensional Systems and Signal Processing 28, 1 (2017), 207–224.
[9]
Yu Liu, Xun Chen, Juan Cheng, and Hu Peng. 2017. A medical image fusion method based on convolutional neural networks. In 2017 20th international conference on information fusion (Fusion). IEEE, Xian,China, 1–7.
[10]
Hikmat Ullah, Basharat Ullah, Longwen Wu, Fakheraldin YO Abdalla, Guanghui Ren, and Yaqin Zhao. 2020. Multi-modality medical images fusion based on local-features fuzzy sets and novel sum-modified-Laplacian in non-subsampled shearlet transform domain. Biomedical Signal Processing and Control 57 (2020), 101724.
[11]
Bin Wang, Jianchao Zeng, Suzhen Lin, and Guifeng Bai. 2019. Multi-band images synchronous fusion based on NSST and fuzzy logical inference. Infrared Physics & Technology 98 (2019), 94–107.
[12]
Yu Zhang, Yu Liu, Peng Sun, Han Yan, Xiaolin Zhao, and Li Zhang. 2020. IFCNN: A general image fusion framework based on convolutional neural network. Information Fusion 54 (2020), 99–118.

Cited By

View all
  • (2024)Medical Surgery Stream Segmentation to Detect and Track Robotic Tools2024 IEEE First International Conference on Artificial Intelligence for Medicine, Health and Care (AIMHC)10.1109/AIMHC59811.2024.00043(194-200)Online publication date: 5-Feb-2024

Index Terms

  1. Multi-scale Feature Fusion Convolutional Neural Network for Multi-Modal Medical Image Fusion

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Other conferences
        CNIOT '23: Proceedings of the 2023 4th International Conference on Computing, Networks and Internet of Things
        May 2023
        1025 pages
        ISBN:9798400700705
        DOI:10.1145/3603781
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 27 July 2023

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. convolutional neural networks
        2. medical image fusion
        3. multi-modal
        4. multi-scale

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        • National Natural Science Foundation of Shanghai
        • National Natural Science Foundation of China

        Conference

        CNIOT'23

        Acceptance Rates

        Overall Acceptance Rate 39 of 82 submissions, 48%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)31
        • Downloads (Last 6 weeks)2
        Reflects downloads up to 16 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Medical Surgery Stream Segmentation to Detect and Track Robotic Tools2024 IEEE First International Conference on Artificial Intelligence for Medicine, Health and Care (AIMHC)10.1109/AIMHC59811.2024.00043(194-200)Online publication date: 5-Feb-2024

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media