Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network

Highlights

• A medical image fusion model of SWT and RBFNN is proposed.

• Proposed prior knowledge-based RBFNN have stronger learning ability.

• The proposed method could effectively fuse detail information of original images.

Abstract

Medical image fusion of images obtained via different modes can expand the inherent information of original images, whereby the fused image has a superior ability to display details than the original sub-images, to facilitate diagnosis and treatment selection. In medical image fusion, an inherent challenge is to effectively combine the most useful information and image details without information loss. Despite the many methods that have been proposed, the effective retention and presentation of information proves challenging. Therefore, we proposed and evaluated a novel image fusion method based on the discrete stationary wavelet transform (DSWT) and radial basis function neural network (RBFNN). First, we analyze the details or feature information of two images to be processed by DSWT by using two-level decomposition to separate each image into seven parts, comprising both high-frequency and low-frequency sub-bands. Considering the gradient and energy attributes of the target, we substituted the pending parts in the same position in the two images by using the proposed enhanced RBFNN. The input, hidden, and output layers of the neural network comprised 8, 40, and 1 neuron(s), respectively. From the seven neural networks, we obtained seven fused parts. Finally, through inverse wavelet transform, we obtained the final fused image. For the neural network training method, the hybrid adaptive gradient descent algorithm (AGDA) and gravitational search algorithm (GSA) were implemented. The final experimental results revealed that the novel method has significantly better performance than the current state-of-the-art methods.

Introduction

Image fusion technology synthesizes multiple source images into a new image This process not only involves superimposition of all image data, but also targeted processing of the target image through one or more algorithms [1], [2], [3]. In medical imaging, the concept of fusion pertains to the combination of two (or more) medical images from different imaging devices by using an algorithm to combine the advantages or complementarities of each image in order to obtain a more informative image. The development of medical imaging technology has resulted in the widespread use of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single-photon emission computed tomography (SPECT), and other imaging techniques in clinical diagnosis and treatment. Different imaging modes can provide different types of medical information to doctors. For instance, CT images have a high spatial resolution, clear bone imaging, and provide a good reference for the positioning of lesions. However, CT images are rather insensitive to adequately display soft tissue or even invasive tumor details. In contrast, MRI soft-tissue imaging provides clear images and is conducive to the determination of the scope of the lesion; similarly, PET and SPECT imaging can provide clear functional information about metabolism within the body. However, due to the low spatial resolution of functional imaging, the diagnosis of pathological tumors can be limited, especially as different imaging principles comprise inherent limitations in acquiring imaging information, and the use of a single type of image would not be optimal for precise visualization. Therefore, combining the advantages of different imaging methods and complementary information through medical image fusion technology can generate sufficient accurate information to facilitate medical diagnosis and treatment planning [4], [5], [6], [7].

In recent decades, research on medical image fusion, especially multiscale transform (MST)-based methods, has gained mainstream popularity. Briefly, the main sequential processes of MST-based fusion methods involve: (1) decomposing the pending source images into several sub-bands, which represent some feature information of the images, by using complicated operators; (2) applying different fusion rules, based on the pixel level, to fuse the corresponding sub-bands together; and (3) inverse transformation to obtain the final fusion image. Early conventional methods included discrete wavelet transform-based methods [1], [8] and Laplacian pyramid transform-based methods [9], [10]. The early methods obtained different image feature information through multiscale transformation because different frequency components are processed by the same or simple fusion rules when the sub-bands are fused together. However, a limitation of these methods is that only one image characteristic is considered, whereas others are ignored, and this can seriously reduce the fusion effect [11]. In order to solve this issue, some enhanced algorithms were proposed. Tian et al. [12] applied different fusion rules to diverse frequency bands based on pixel or regional information. Xu et al. [13] proposed the fractional wavelet transform method, which can better define fusion coefficients. Similarly, Lie et al. proposed an MST-based sparse representation theory [14]. Although the abovementioned methods adopt different rules in various frequency bands to obtain good results, these methods cannot optimally fuse details and contour information due to the limitations of their respective established rules. Moreover, because of the down-sampling and up-sampling involved, artifact-related problems are sometimes unavoidable. Furthermore, image fusion methods, based on stationary wavelet transform (SWT), were proposed to effectively avoid the Gibbs phenomenon [15], [16], although the imperfect formulation of rules hampered optimal image fusion. In addition, other shift-invariant-based methods, such as nonsubsampled contourlet transform (NSCT), have garnered interest. Based on the NSCT domain, Ganasala et al. [17] applied the entropy and Laplacian operators to the corresponding low- and high-frequency sub-bands, respectively. NSCT-based methods have implemented rules based on phase congruency, directive contrast-based rules, or Laplacian energy in order to merge the corresponding low- and high-frequency domains [18], [19]. The non-subsampled shearlet transform (NSST) method, which applies a shear-wave filter to decompose and reconstruct pending images, constitutes another mainstream shift-invariant-based method. Liu et al. [20] consider the gradient factor to optimize the problem of coefficients based on the structure tensor and NSST. To improve the directional information, Singh et al. [21] adopted ripplet transform and NSST to design a cascaded model. The above-described methods have achieved accurate and detailed visualization of directional tissue features. However, due to the large difference in the parameter performance of different filters, the stability of the fusion effect is greatly affected by the selection of parameters. In addition, computational complexity affects fusion efficiency [22], [23].

Furthermore, neural network-based methods are applied to medical image fusion. The pulse-coupled neural network (PCNN) is a simplified neural network model that is based on the principles underlying cat vision. The application of the PCNN-based method – processed under the global domain – to the field of image fusion can facilitate the retention of more detailed information. However, the effective stimulation of neurons to maximize their effect without training remains a challenge. Huang et al. [24] add the Laplacian energy of the image block as input stimulation for each neuron in the PCNN. To improve the performance of the PCNN to increase the quality of the fused image, parameter-adaptive PCNN methods were previously applied to the high-frequency band of the NSST domain [25], [26]. Despite the availability of many PCNN-based fusion algorithms, the optimization of coefficients and the threshold definition are still being investigated. In recent years, convolutional neural network (CNN)-based methods have emerged as a popular approach for solving problems pertaining to medical image fusion [27], [28], [29]. However, the difficulty of training deep neural networks and the availability of small-sample training data have hampered the sustained performance of the neural network. Based on the multiscale domain, the sampling or convolution operation in CNN can easily cause information loss during the fusion process. We previously proposed a medical image-fusion method that was based on a fuzzy radial basis function neural network [30], but only considered the pixel features based on the image domain to stimulate the input neurons, which may lead to the insufficiency of the neural network’s cognitive ability and the lack of optimal performance.

At present, the main research direction of medical image fusion is to establish specific fusion rules on each frequency sub-band based on shift-invariant-based multiscale transforms. However, the identification of the most suitable fusion rules is problematic. To address this gap, in the present study, we proposed and tested a novel medical image fusion method based on the discrete stationary wavelet transform (DSWT) and radial basis function neural network (RBFNN).