Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network
Introduction
Image fusion technology synthesizes multiple source images into a new image This process not only involves superimposition of all image data, but also targeted processing of the target image through one or more algorithms [1], [2], [3]. In medical imaging, the concept of fusion pertains to the combination of two (or more) medical images from different imaging devices by using an algorithm to combine the advantages or complementarities of each image in order to obtain a more informative image. The development of medical imaging technology has resulted in the widespread use of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single-photon emission computed tomography (SPECT), and other imaging techniques in clinical diagnosis and treatment. Different imaging modes can provide different types of medical information to doctors. For instance, CT images have a high spatial resolution, clear bone imaging, and provide a good reference for the positioning of lesions. However, CT images are rather insensitive to adequately display soft tissue or even invasive tumor details. In contrast, MRI soft-tissue imaging provides clear images and is conducive to the determination of the scope of the lesion; similarly, PET and SPECT imaging can provide clear functional information about metabolism within the body. However, due to the low spatial resolution of functional imaging, the diagnosis of pathological tumors can be limited, especially as different imaging principles comprise inherent limitations in acquiring imaging information, and the use of a single type of image would not be optimal for precise visualization. Therefore, combining the advantages of different imaging methods and complementary information through medical image fusion technology can generate sufficient accurate information to facilitate medical diagnosis and treatment planning [4], [5], [6], [7].
In recent decades, research on medical image fusion, especially multiscale transform (MST)-based methods, has gained mainstream popularity. Briefly, the main sequential processes of MST-based fusion methods involve: (1) decomposing the pending source images into several sub-bands, which represent some feature information of the images, by using complicated operators; (2) applying different fusion rules, based on the pixel level, to fuse the corresponding sub-bands together; and (3) inverse transformation to obtain the final fusion image. Early conventional methods included discrete wavelet transform-based methods [1], [8] and Laplacian pyramid transform-based methods [9], [10]. The early methods obtained different image feature information through multiscale transformation because different frequency components are processed by the same or simple fusion rules when the sub-bands are fused together. However, a limitation of these methods is that only one image characteristic is considered, whereas others are ignored, and this can seriously reduce the fusion effect [11]. In order to solve this issue, some enhanced algorithms were proposed. Tian et al. [12] applied different fusion rules to diverse frequency bands based on pixel or regional information. Xu et al. [13] proposed the fractional wavelet transform method, which can better define fusion coefficients. Similarly, Lie et al. proposed an MST-based sparse representation theory [14]. Although the abovementioned methods adopt different rules in various frequency bands to obtain good results, these methods cannot optimally fuse details and contour information due to the limitations of their respective established rules. Moreover, because of the down-sampling and up-sampling involved, artifact-related problems are sometimes unavoidable. Furthermore, image fusion methods, based on stationary wavelet transform (SWT), were proposed to effectively avoid the Gibbs phenomenon [15], [16], although the imperfect formulation of rules hampered optimal image fusion. In addition, other shift-invariant-based methods, such as nonsubsampled contourlet transform (NSCT), have garnered interest. Based on the NSCT domain, Ganasala et al. [17] applied the entropy and Laplacian operators to the corresponding low- and high-frequency sub-bands, respectively. NSCT-based methods have implemented rules based on phase congruency, directive contrast-based rules, or Laplacian energy in order to merge the corresponding low- and high-frequency domains [18], [19]. The non-subsampled shearlet transform (NSST) method, which applies a shear-wave filter to decompose and reconstruct pending images, constitutes another mainstream shift-invariant-based method. Liu et al. [20] consider the gradient factor to optimize the problem of coefficients based on the structure tensor and NSST. To improve the directional information, Singh et al. [21] adopted ripplet transform and NSST to design a cascaded model. The above-described methods have achieved accurate and detailed visualization of directional tissue features. However, due to the large difference in the parameter performance of different filters, the stability of the fusion effect is greatly affected by the selection of parameters. In addition, computational complexity affects fusion efficiency [22], [23].
Furthermore, neural network-based methods are applied to medical image fusion. The pulse-coupled neural network (PCNN) is a simplified neural network model that is based on the principles underlying cat vision. The application of the PCNN-based method – processed under the global domain – to the field of image fusion can facilitate the retention of more detailed information. However, the effective stimulation of neurons to maximize their effect without training remains a challenge. Huang et al. [24] add the Laplacian energy of the image block as input stimulation for each neuron in the PCNN. To improve the performance of the PCNN to increase the quality of the fused image, parameter-adaptive PCNN methods were previously applied to the high-frequency band of the NSST domain [25], [26]. Despite the availability of many PCNN-based fusion algorithms, the optimization of coefficients and the threshold definition are still being investigated. In recent years, convolutional neural network (CNN)-based methods have emerged as a popular approach for solving problems pertaining to medical image fusion [27], [28], [29]. However, the difficulty of training deep neural networks and the availability of small-sample training data have hampered the sustained performance of the neural network. Based on the multiscale domain, the sampling or convolution operation in CNN can easily cause information loss during the fusion process. We previously proposed a medical image-fusion method that was based on a fuzzy radial basis function neural network [30], but only considered the pixel features based on the image domain to stimulate the input neurons, which may lead to the insufficiency of the neural network’s cognitive ability and the lack of optimal performance.
At present, the main research direction of medical image fusion is to establish specific fusion rules on each frequency sub-band based on shift-invariant-based multiscale transforms. However, the identification of the most suitable fusion rules is problematic. To address this gap, in the present study, we proposed and tested a novel medical image fusion method based on the discrete stationary wavelet transform (DSWT) and radial basis function neural network (RBFNN).
Section snippets
Study design
Considering the shift invariance and the number of calculations, we selected DSWT as the multiscale transform operator. Next, we performed two-level wavelet decomposition to obtain 14 sub-bands representing different information features of two source images that are to be fused. For the corresponding pair of sub-bands, fully considering the pixel characteristics and the interaction between pixels, we established a radial basis neural network that includes 8, 40, and 1 neuron(s) in the input,
Results of visual observation
For each type of dataset pair to be fused, we took two fused images. The fusion performances of six pairs of datasets based on seven different methods are shown in Fig. 5, Fig. 6, Fig. 7, Fig. 8, Fig. 9, Fig. 10. In these figures, (a) and (b) show the two source images to be fused; (c)–(i) represent the fused images obtained by the WT, NSCT, SR, APCNN, CNN, and FRBFNN-based methods as well as the proposed method, respectively. First, we observed the CT-MRI datasets (CT-MRI dataset 1 and CT-MRI
Discussion
Medical image fusion is used to complementarily combine medical images obtained through different imaging modalities. By organically combining anatomical information or functional information, the fused image displays comprehensive information based on multiple single-mode imaging sources without loss of information or energy in the single-mode source images. The WT- and NSCT-based methods obviously failed to achieve the desired fusion effects. MST-based methods are increasingly emerging as
Conclusion
The proposed hybrid model based on DSWT and enhanced RBFNN captures and processes the details of each sub-band and overcomes the limitations that afflict previously established medical image fusion methods. Additionally, we apply an adaptive choice approach based on other competitive algorithms to create teacher data, making the proposed structure learn the strengths from other methods. In comparative experiments using abundant data on several mainstream methods, the proposed method could not
CRediT authorship contribution statement
Zhen Chao: Conceptualization, Methodology, Software, Writing – original draft. Xingguang Duan: Investigation. Shuangfu Jia: Assessment. Xuejun Guo: Validation. Hao Liu: Validation, Supervision. Fucang Jia: Supervision, Writing – reviewing and editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
We are very grateful for the open-source code shared [14], [26], [27]. In addition, for Refs. [1], [17], we reproduced code according to the papers’ contents. The codes are available on https://github.com/med-img/dwt_based_img_fusion.git and https://github.com/med-img/NSCT_Padma_2014_img_fusion.git.
Funding
This work was supported by the National Natural Science Foundation of China (grant nos. 62172401, 82001905, and 12026602), and was partly supported by the National Key Research and Development
References (44)
- et al.
A wavelet-based image fusion tutorial
Pattern Recognit.
(2004) - et al.
Infrared and visible image fusion methods and applications: A survey
Inf. Fusion
(2019) - et al.
Medical image fusion: A survey of the state of the art
Inf. Fusion
(2014) - et al.
Medical image fusion using discrete fractional wavelet transform
Biomed. Signal Process. Control
(2016) - et al.
A general framework for image fusion based on multi-scale transform and sparse representation
Inf. Fusion
(2015) - et al.
Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion
Neurocomputing
(2017) - et al.
Infrared and visible image fusion based on visual saliency map and weighted least square optimization
Infrared Phys. Technol.
(2017) - et al.
Multi-focus image fusion using pulse coupled neural network
Pattern Recognit. Lett.
(2007) - et al.
Multi-focus image fusion with a deep convolutional neural network
Inf. Fusion
(2017) - et al.
Multi-modality image fusion based on enhanced fuzzy radial basis function neural networks
Phys. Med.
(2018)
R-peaks detection based on stationary wavelet transform
Comput. Methods Programs Biomed.
Application of radial basis function artificial neural network to quantify interfacial energies related to membrane fouling in a membrane bioreactor
Bioresour. Technol.
Prediction of interfacial interactions related with membrane fouling in a membrane bioreactor based on radial basis function artificial neural network (ANN)
Bioresour. Technol.
An entropy-based evaluation method for knowledge bases of medical information systems
Expert Syst. Appl.
Image fusion with guided filtering
IEEE Trans. Image Process.
An overview of multi-modal medical image fusion
Neurocomputing
Fast curvelet transform through genetic algorithm for multimodal medical image fusion
Soft Comput.
Current trends in medical image registration and fusion
Egypt. Inform. J.
Medical image of PET/CT weighted fusion based on wavelet transform
The Laplacian pyramid as a compact image code
IEEE Trans. Commun.
Multi-scale fusion algorithm comparisons: Pyramid, DWT and iterative DWT
Medical image fusion via an effective wavelet-based approach
EURASIP J. Adv. Signal Process.
Cited by (28)
Learnable bilevel optimization method for electrical capacitance tomography
2024, Signal ProcessingAn efficient approach to medical image fusion based on optimization and transfer learning with VGG19
2024, Biomedical Signal Processing and ControlGaussian RBFNN method for solving FPK and BK equations in stochastic dynamical system with FOPID controller
2023, International Journal of Non-Linear MechanicsParameter adaptive unit-linking pulse coupled neural network based MRI–PET/SPECT image fusion
2023, Biomedical Signal Processing and ControlMSE-Fusion: Weakly supervised medical image fusion with modal synthesis and enhancement
2023, Engineering Applications of Artificial IntelligenceCitation Excerpt :However, they lose the targeting of key features in the feature extraction process (Li et al., 2020), and it is difficult to fully characterize the deep-structure and shallow-detail information in the fused images. In recent years, deep learning has been widely used in image processing fields, such as modal synthesis (Tie et al., 2020; Sivanesan et al., 2021; Liu et al., 2021a,b; Reaungamornrat et al., 2022; Gao et al., 2022a), image fusion (Manoj et al., 2021a; Wang et al., 2022a; Zhen et al., 2022; Wang et al., 2022b; Liang and Xu, 2022; Manoj et al., 2021b; Wang et al., 2022c; Liu et al., 2022), and image segmentation (Srivastava et al., 2022; Wang et al., 2022d; Karthik et al., 2021; Li et al., 2022; Ouyang et al., 2022) because of its excellent feature extraction ability, and has yielded good performance. Convolutional neural network (CNN) as a representative method of deep learning (Esteva et al., 2017; Kumar et al., 2020; Prabhishek and Achyut, 2021; Ma et al., 2022a), is playing an increasingly important role in image fusion (Dian et al., 2020; Dong et al., 2022).
Combining spectral total variation with dynamic threshold neural P systems for medical image fusion
2023, Biomedical Signal Processing and ControlCitation Excerpt :The final step is to transform the synthesized components into the spatial domain. Some popular transformation methods that have been applied to the problem of medical image synthesis can be mentioned, such as LP transform [13,15], discrete stationary wavelet transform (DSWT) [16,17], NSCT [2,18], and NSST [11,19,20]. Commonly used methods to synthesize components on the transform domain can be listed as Min–Max selection [21], averaging rules [22,23], local energy function maximization [24,25], and sum-modified-Laplacian (SML) [26].