A novel similarity based quality metric for image fusion

doi:10.1016/j.inffus.2006.09.001

Information Fusion

Volume 9, Issue 2, April 2008, Pages 156-160

https://doi.org/10.1016/j.inffus.2006.09.001 Get rights and content

Abstract

A novel objective quality metric for image fusion is presented. The interest of our metric lies in the fact that the redundant regions and the complementary/conflicting regions are treated respectively according to the structural similarity between the source images. The experiments show that the proposed measure is consistent with human visual evaluations and can be applied to evaluate image fusion schemes that are not performed at the same level.

Introduction

Quality assessment of different image fusion schemes is traditionally carried out by subjective evaluations [1]. However, they are not only tedious, slow, expensive, difficult to reproduce and verify, but they are also impossible to be embedded into the rendering image fusion algorithms to optimize the parameter settings. Hence, although it cannot be denied that subjective tests are important in characterizing fusion performance, objective image fusion performance metrics that are consistent with human visual perception appear as a valuable alternative. Several schemes have been proposed for the development of performance metrics. Li et al. [2] associated image fusion performance with the standard deviation of the difference image between the fused image and the reference image, which was created by a simple cut and paste process. However, it is impossible to get the ideal fusion image in most real world applications. Therefore, some non-reference image fusion quality assessments have been proposed. Mutual information (MI) given by Qu et al. [3] is the most commonly used information-theory related metric, but it can not correctly assess the performance of image fusion schemes that are not performed at the same level [3]. Xydeas and Petrović [4] evaluated the fusion performance by calculating edge information preservation values in the fused image. Recently, objective image fusion performance measures based on the structural similarity metric proposed by Wang et al. [8] have come into being [5], [6], [7].

The performance metric, which is calculated pixel by pixel or region by region, given in [5] are the weighted average of the similarities between the fused image and each of the source images. It is not suitable for the evaluation of regions which contain complementary or conflicting information in the source images, where one of them should be selected to comprise the fused image in a good grayscale image fusion scheme.

Given two source images and a single fused image, in this paper, we propose a novel objective image fusion performance metric, in which complementary or conflicting regions are distinguished from redundant regions in the two source images using the structural similarity image quality measure proposed in [8] and treated separately. Thus, the results of the proposed measure are more consistent with subjective evaluations.

Section snippets

The structural similarity metric by Wang and Bovik

The structural similarity (SSIM) metric introduced by Wang and Bovik [8] for the corresponding regions in a reference original signal x and the test image signal y is defined as $SSIM (x, y | w) = \frac{(2 {\bar{w}}_{x} {\bar{w}}_{y} + C_{1}) (2 σ_{w_{x} w_{y}} + C_{2})}{({\bar{w}}_{x}^{2} + {\bar{w}}_{y}^{2} + C_{1}) (σ_{w_{x}}^{2} + σ_{w_{y}}^{2} + C_{2})}$ which can be decomposed as $SSIM (x, y | w) = \frac{(2 {\bar{w}}_{x} {\bar{w}}_{y} + C_{1}) (2 σ_{w_{x}} σ_{w_{y}} + C_{2}) (σ_{w_{x} w_{y}} + C_{3})}{({\bar{w}}_{x}^{2} + {\bar{w}}_{y}^{2} + C_{1}) (σ_{w_{x}}^{2} + σ_{w_{y}}^{2} + C_{2}) (σ_{w_{x}} σ_{w_{y}} + C_{3})}$ where C₁, C₂ and C₃ are small constants, with C₃ = C₂/2, w_x denotes the sliding window or the region under consideration in x, ${\bar{w}}_{x}$ is the mean of w_x, $σ$

The new image fusion quality metric

The goal of image fusion is to increase completeness by integrating complementary information into the fused image and to eliminate conflicting information. Therefore a good fusion performance measure should correctly estimate how much information is preserved in the fused image, especially for the regions that contain complementary or conflicting information in the source images.

Generally speaking, for regions where the two source images contain conflicting information, a good fusion scheme

Experimental results

To verify the performance of the proposed approach, we have carried out two experiments, in both of which four image fusion schemes, the Laplacian pyramid [9], the discrete wavelet transform (DWT) [10], the ratio pyramid [11] and the average method, are tested. For the first three fusion schemes, a 3-level decomposition was performed, with the approximation coefficients of the two input images averaged and the larger absolute values of the high subbands selected.

In the first experiment, 32 sets

Conclusions

In this paper, we have proposed an objective image fusion performance metric that performs different operations when evaluating different local regions according to the similarity level between the source images. Our proposed method is consistent with the human vision perception. In particular, our metric gives good results when evaluating different fusion schemes that are not performed at the same level.

There are several areas in which our quality metric can be improved. Entropy or mutual

References (11)

A. Toet et al.
Perceptual evaluation of different image fusion schemes
Displays
(2003)
A. Toet
Image fusion by a ratio of low-pass pyramid
Pattern Recognition Letters
(1989)
H. Li, B.S. Manjunath, S.K. Mitra, Multi-sensor image fusion using the wavelet transform, in: Proceedings of the IEEE...
G.H. Qu et al.
Information measurement for performance of image fusion
Electronics Letters
(2002)
V. Xydeas et al.
Objective image fusion performance measure
Electronics Letters
(2000)

There are more references available in the full text version of this article.

Cited by (410)

FATFusion: A functional–anatomical transformer for medical image fusion
2024, Information Processing and Management
Medical image fusion (MIF) strives to obtain an informative fused image by integrating complementary information, e.g., functional metabolic properties and anatomical soft tissues, to facilitate disease diagnosis and treatment planning. To solve the problems of existing MIF algorithms, including inconvenient and manually designed transformation methods, complicated fusion strategies, global complementary information loss, and insufficient consideration of the modality-specific intrinsic characteristics, in this paper, we innovatively propose a Functional–Anatomical Transformer for MIF named FATFusion. In the proposed method, a functional multiscale branch (FMB) and an anatomical multiscale branch (AMB) are designed to extract interscale characteristics belonging to each modality. In addition, functional-guided transformer modules (FGTMs) and anatomical-guided transformer modules (AGTMs) are elaborated to communicate and gather the perceived features for further global complementary information interaction and aggregation. Moreover, the pixel loss and total variation loss are designed to train FATFusion in an end-to-end and unsupervised manner, guiding the deep fusion model to leverage more significant properties. Promising quantitative and qualitative assessments illustrate that FATFusion surpasses alternative state-of-the-art MIF methodologies, excelling in both objective evaluation and perceptible observation. Furthermore, successful generalization results reveal that FATFusion has impressive generalization ability. The source code of the proposed method is available at https://github.com/tthinking/FATFusion.
Multi-focus image fusion via interactive transformer and asymmetric soft sharing
2024, Engineering Applications of Artificial Intelligence
Multi-focus image fusion technology is wildly used in the digital photography, and also be considered as a pre-task of other high level vision tasks. The main purpose is producing an all-in-focus image from multiple partly focused sources accurately, naturally and efficiently. Nowadays, transformer models have achieved great success in numerous vision-related tasks, and its powerful attention modeling ability brings significant possibilities for focus property detection. To make multi-focus image fusion technology even further, in this paper, a novel multi-focus image fusion method using interactive transformer and asymmetric soft sharing is proposed. First, for taking the transformer’s advantages in global context modeling and improve its limitations in diversity and efficiency, a locally-enhanced interactive approach is devised. More specifically, it used cross-scale and cross-domain computation strategy to overcome the transformer’s certain limitations when comes to domain-specific task, and simultaneously alleviate the insufficient local feature perception and redundant computational cost of existing approach. Second, to cope with the problems of fusion image distortion and artifacts, the proposed method adopts a multi-task learning strategy with asymmetric soft sharing. It predicts the fusion images and decision maps at the same time for avoiding image distortion and further obtaining natural fusion effect. Experimental results reveal that, the proposed method reaches promising focus property detection performance and high-fidelity fusion results. Moreover, when compared with other state-of-the-art methods on five multi-focus image datasets, it is more prominent in both qualitative and quantitative analysis as well as efficiency. The source code is available at https://github.com/zwy0913/MFFT.
New insights into multi-focus image fusion: A fusion method based on multi-dictionary linear sparse representation and region fusion model
2024, Information Fusion
This paper proposes a method based on multi-dictionary linear sparse representation and region fusion model for achieving high-quality multi-focus image fusion (MFIF). First, a multi-dictionary linear sparse representation model (MDLSR) is proposed for focus detection of source images, in which the focus information of the source image is characterized using linear coefficients between the dictionary and the input image, while the dictionary is a set of images including the source images. MDLSR exploits the correlation of feature information among local neighborhoods of the source image, and this feature significantly improves the accuracy of focus detection. Next, we propose an edge-weighting closed-form matting (EWCM) to define the source image focus boundary accurately. EWCM calculates the weights at each position in a content-adaptive manner, suppressing the effects of vignetting artifacts near the edges to preserve more edge details. Then, to avoid the focused/defocused boundary (FDB) region fusion problem caused by the defocus spread effect (DSE), we propose a region fusion model (RFM) for the MFIF, which uses correction and compensation to mitigate this problem. Finally, we also adopt an iterative update approach to improve the quality of the fused images. These designs achieve a win-win situation regarding fusion performance and realization ability. The experimental results on four typical datasets show that our method outperforms 37 state-of-the-art methods in terms of subjective visual effect, objective metric analysis and generalization ability. Our code is available at https://github.com/JWatawang/MDLSR_RFM.
Infrared-visible image fusion method based on multi-scale shearing Co-occurrence filter
2024, Infrared Physics and Technology
Infrared-visible image fusion aims to achieve salient targets and abundant texture details. However, most of fusion methods based on multi-scale decomposition fail to effectively preserve detailed and texture information. To avoid these issues, a novel fusion method is proposed by means of multi-scale shearing Co-occurrence filter (MSCOF). This multi-scale decomposition tool possesses the flexible multi-direction and shift-invariant properties of the Shearlet, as well as the smoothing advantages of the Co-occurrence filter (COF). In this study, we obtain approximate and detail subbands by performing MSCOF decomposition on the original images. A weighted fusion rule based on the saliency intensity value (SIV) is performed to combine approximate subbands. Notably, COF is employed on the saliency intensity maps to better preserve edge information of the original image on the fused approximate subbands. Meanwhile, the improved local gradient energy (ILGE) method is utilized to fuse the detail subbands. The results of experimental analysis fully prove that our method is reliable and feasible, and that the obtained objective evaluation metrics are superior to those of some typical methods.
AFCANet: An adaptive feature concatenate attention network for multi-focus image fusion
2023, Journal of King Saud University - Computer and Information Sciences
For multi-focus image fusion, the existing deep learning based methods cannot effectively learn the texture features and semantic information of the source image to generate high-quality fused images. Thus, we develop a new adaptive feature concatenate attention network named AFCANet, which adaptively learns cross-layer features and retains the texture features and semantic information of images to generate visually appealing fully focused images. In AFCANet, the encoder-decoder network is used as the backbone network. In the unsupervised training stage, an adaptive cross-layer skip connection mode is designed, and a cross-layer adaptive coordinate attention module is built to acquire meaningful information from the image along with ignoring unimportant information to obtain a better image fusion effect. In addition, in the middle of the encoder-decoder network, we also introduce an effective channel attention module to fully learn the output of the encoder, and accelerate network convergence. In the inference stage, we apply the pixel-based spatial frequency fusion rules to fuse the adaptive features learned by the encoder, which can successfully combine the texture and semantic information of the image and produce a more precise decision map. Extensive experiments on public datasets and the HBU-CVMDSP dataset show that our AFCANet can effectively improve the accuracy of the decision map in the focus and defocus regions, as well as improve the ability to retain the abundant details and edge features of the source image.
Infrared and visible image fusion method based on principal component analysis network and multi-scale morphological gradient
2023, Infrared Physics and Technology
In infrared (IR) and visible image fusion, energy conservation and detail extraction are two key problems. We propose a novel IR and visible image fusion model based on principal component analysis network (PCANet) and multi-scale morphological gradient (MSMG), aiming to better preserve energy and extract details. Firstly, we obtain the features of IR and visible images through PCANet. The features obtained by PCANet have more powerful representation capabilities in IR target perception and visible detail description. Secondly, we develop a fusion strategy combining MSMG with guided filter to obtain the corresponding feature map weights. This fusion strategy is able to preserve more IR image energy and extract more visible image details. Finally, a weighted-averaging strategy is used to obtain the fused image. The effectiveness of the proposed method is verified by two datasets with more than 80 pairs of source images in total. Compared with 17 representative methods, the experimental results demonstrate that the proposed method can achieve the state-of-the-art in both visual quality and objective evaluation.

View all citing articles on Scopus

^☆: This work is supported by the Research Fund for the Doctoral Program of Higher Education (No. 20030701003) and the National Natural Science Foundation of China (No. 60477038).

View full text

A novel similarity based quality metric for image fusion☆

Abstract

Introduction

Section snippets

The structural similarity metric by Wang and Bovik

The new image fusion quality metric

Experimental results

Conclusions

Displays

Pattern Recognition Letters

Information measurement for performance of image fusion

Electronics Letters

Objective image fusion performance measure

Electronics Letters