Efficient fused convolution neural network (EFCNN) for feature level fusion of medical images

Abstract

This paper proposes an Efficient Fused Convolution Neural Network (EFCNN) for feature-level fusion of medical images. The proposed network architecture leverages the strengths of both deep Convolution Neural Networks (CNNs) and fusion techniques to achieve improved efficiency in medical image fusion. Image fusion of CT and MRI images can help medical professionals to make more informed diagnosis, plan more effective treatments, and ultimately improve patient outcomes. Recently many researchers are working to develop efficient medical fusion technique. To contribute to this field, authors have attempted to fuse images at feature level using Bilinear Activation Function (BAM) for feature extraction and softmax based Soft Attention (SA) fusion rule for fusion. The EFCNN model uses a two-stream CNN architecture to process input images, which are then fused at the feature level using an attention mechanism. The proposed approach is evaluated on Whole Brain Atlas Harvard dataset. The EFCNN model demonstrated superior performance in various performance indices, including ISSIM, MI, and PSNR, with respective values of 0.41, 4.42, and 57.21 when SA was utilized. Furthermore, the proposed model exhibited favourable performance in terms of Spatial Frequency, Average Gradient, and Edge-intensity, with corresponding values of 57.3, 16.83, and 157.72 on a medical dataset when EFCNN was applied without SA fusion. However, subjective evaluation indicated that images were improved with SA fusion. These results indicate that the EFCNN model surpasses state-of-the-art methods. An exhaustive ablation study was conducted to investigate the efficacy of the proposed model, which further confirmed its accuracy. The significance of this work is its potential implications for medical diagnosis and treatment planning, where precise and efficient image analysis is crucial.

Appendices

Appendix

A Performance measure

The five evaluation metrics are used as performance measurement which are Mutual Information (MI) [31], Spatial frequency [33] and Average Gradient [34], Edge-intensity [35], and Entropy [36, 37]. In information theory mutual information (MI) represents statistical dependency of one variable on other. If X and Y are two source images and F the fused image, the MI can be calculated according to the formula given by (3) [31].

$$\begin{aligned} M^{XY}_{F}={I_{FX}(f,x)}+{I_{FY}(f,y)}. \end{aligned}$$

(3)

Where $I_{FX}(f,x)$ and $I_{FY}(f,y)$ represents amount of information F contains about X and Y respectively.Which can be calculated as given by (4 and 5).

$$\begin{aligned} I_{FX}(f,x)=\sum _{f,x}P_{FX}(f,x)log{\frac{P_{FX}(f,x)}{P_{F}(f)P_{X}(x)}} \end{aligned}$$

(4)

$$\begin{aligned} I_{FY}(f,y)=\sum _{f,y}P_{FY}(f,y)log{\frac{P_{FY}(f,y)}{P_{F}(f)P_{Y}(y)}} \end{aligned}$$

(5)

Here $P_{F}(f)$, $P_{Y}(y)$ & $P_{X}(x)$ represents marginal probability distributions and $P_{FX}(f,x) and P_{FY}(f,y)$ represents joint probability distribution of fused and source images. Firstly [38], proposed an evaluation metric showing structural similarity of source images, later [30] proposed an improved version of this metric i.e. ISSIM shown by the equation given below (6), successfully distinguishing conflicted and complementary information from redundant regions.

$$\begin{aligned} ISSIM(x,y,f(\mid )w)={\left\{ \begin{array}{ll}\lambda (w)SSIM(x,f(\mid )w)+(1-\lambda (w)SSIM(y,f(\mid )w), &{} \\ for SSIM(x,y(\mid )w\ge 0.75 &{} \\ max(SSIM(x,f(\mid )w),SSIM(y,f(\mid )w))&{} \\ for SSIM(x,y(\mid )w)< 0.75\end{array}\right. } \end{aligned}$$

(6)

Where $\lambda (w)$ is local weight and s(x($\mid$)w) and s(y($\mid$)w) are variances of $w_{x}$ and $w_{y}$ respectively.

Spatial frequency (SF) [39],given by equation 6 represents the overall active level of the image.The metric selects the region with maximum spatial frequency by segregating the image by normalized cuts, and combining all such regions to reconstruct the final image [33]. Equations 7 and 8 shows the row (RF) and column frequency (CF) of the image.

$$\begin{aligned} RF=\sqrt{\frac{1}{XY}\times \sum _{x=0}^{X-1} \sum _{y=1}^{Y-1}[F(x,y)-F(x,y-1)]^2} \end{aligned}$$

(7)

$$\begin{aligned} CF=\sqrt{\frac{1}{XY}\times \sum _{y=0}^{Y-1}\sum _{x=1}^{X-1}[F(x,y)-F(x-1,y)]^2} \end{aligned}$$

(8)

$$\begin{aligned} SF=\sqrt{(RF)^{2}+(CF)^{2}} \end{aligned}$$

(9)

Average Gradient (AG) [34] given by (10). A higher value of AG represent more accurate edge details.

$$\begin{aligned} AG=\frac{1}{(M-1)(N-1)}\times \sum _{m=1}^{M-1}\sum _{n=1}^{N-1}(\frac{1}{4})\sqrt{{\frac{[\partial (m,n)}{\partial m]^2}}+{\frac{[\partial (m,n)}{\partial n]^2}}} \end{aligned}$$

(10)

Where (m, n) shows image co-ordinate and their gradients are horizontal and vertical gradients respectively. Peak signal to Noise Ratio (PSNR) is given by (11) [32].

$$\begin{aligned} PSNR=20log_{10}{\frac{L^2}{\frac{1}{MN} \sum _{i=1}^{M} \sum _{j=1}^{N} (I_{r}(i,j)-I_{f}(i,j))^2 }} \end{aligned}$$

(11)

A high value of PSNR indicates good results.

A higher value of edge intensity given by (12) [40] shows clearer fused image. The edge intensity can be calculated using Sobel operator.

$$\begin{aligned} EI=\sqrt{(S_{x})^2+(S_{y})^2} \end{aligned}$$

(12)

where

$$\begin{aligned} S_{x}=f*h_{x} \end{aligned}$$

(13)

$$\begin{aligned} S_{y}=f*h_{y} \end{aligned}$$

(14)

$$\begin{aligned} h_{x}= \left( \begin{array}{ccc} 1&{}0&{}1\\ -2&{}0&{}2\\ -1&{}0&{}1 \end{array}\right) \end{aligned}$$

(15)

$$\begin{aligned} h_{x}= \left( \begin{array}{ccc} -1&{}-2&{}-1\\ 0&{}0&{}0\\ 1&{}2&{}1 \end{array}\right) \end{aligned}$$

(16)

Another performance measure Entropy (EN) has been widely used in remote sensing applications to set out the information content of an image [36]. Shannon and Claude [41] were first to propose the use of entropy for estimating information content. The original shannon’s equaton is given by (17).

$$\begin{aligned} H=-\sum _{i=1}^{n} P(K_{i}log_{2}P(K_{i}) \end{aligned}$$

(17)

where i is the no. of messages in a set. and $P(K_{i})$ is probability of transpiring a message. The same theory that is applied on messages can be applied on images too if distinct messages are considered as pixels and message set by 2D image. It assumed that the fused image will have more gray level than the individual one as more the amount of information more will be the gray levels. But often it is not the information but the noise that is contributing towards the increased number of gray levels[36].