A feature-based metric for the quantitative evaluation of pixel-level image fusion

Abstract

Pixel-level image fusion has been investigated in various applications and a number of algorithms have been developed and proposed. However, few authors have addressed the problem of how to assess the performance of those algorithms and evaluate the resulting fused images objectively and quantitatively. In this study, two new fusion quality indexes are proposed and implemented through using the phase congruency measurement of the input images. Therefore, the feature-based measurements can provide a blind evaluation of the image fusion result, i.e. no reference image is needed. These metrics take the advantage of the phase congruency measurement which provides a dimensionless contrast- and brightness-invariant representation of image features. The fusion quality indexes are compared with recently developed blind evaluation metrics. The validity of the new metrics are identified by the test on the fusion results achieved by a number of multiresolution pixel-level fusion algorithms.

Introduction

The analysis of multisensor images benefits from the technique named image fusion. The applications include but are not limited to military target detection, medical imaging, remote sensing, nondestructive inspection, and intelligent surveillance [1]. The study of fusing multiple images mainly focuses on how to optimize the use of the redundant and complementary information provided by heterogeneous image sensors. With different purposes, the image fusion algorithms can be classified as either combination or classification fusion. In the first case the fusion algorithm consists of combining the complementary features from multiple input images while in the second case the redundant information is mainly used for making a decision through modeling. The output of the first type of image fusion is still an image but comprised of the most salient features captured from the different sensors. Usually such fusion operation is implemented with four major steps at the pixel-level as illustrated in Fig. 1. The success of a specific post-processing or analysis relies largely on the performance of the specific fusion algorithm. The goal of the classification fusion is to derive a thematic map that indicates certain homogeneous characteristics of pixel regions in the image. This process needs a higher-level operation like feature- or decision-level fusion. In order to determine the performance of different fusion algorithms, an objective metric that can quantitatively measure the quality of the results should be introduced. However, such a metric would largely depend on the requirements of the specific application. A straightforward way to implement the evaluation is through the comparison with a reference image, which is assumed to be perfect. The metrics for image comparison are often employed for the quantitative assessment of image fusion. In the case of classification fusion, the resulting thematic map is compared with the ground truth data. The classification results are then used to generate a confusion matrix. Alternatively the classification errors for each of the classes, and for various thresholds, can also be represented by a receiver operating characteristic (ROC) curve. The perfect results can be prepared with the help of experts’ experience. Unfortunately, the reference image is not always perfect or available practically, thus, raising the need for the quantitative and blind evaluation of the fused images.

A typical example for pixel-level image fusion is the fusion of multi-focus images from a digital camera [2], [3]. In such case, a cut and paste operation is applied to obtain the full-focus image that will serve as a reference for evaluating the fusion results. However, such operation does not assure a perfect reference image. In some applications, the ground truth reference can be generated from a more precise measurement. For example, in nondestructive testing the thickness of a multilayer-structured specimen from an aircraft is estimated from the eddy current inspection or the ultrasonic testing. The actual thickness map of the specimen can be obtained through the post-teardown inspection with X-ray, a destructive process that is used to verify the accuracy of the estimation [4]. Such comparison can only be applied after the acquired images are fully registered, i.e. converted to the same resolution, size, and format. The evaluation metric should be optimized for the image feature. Pixel-by-pixel comparison does not meet the requirement, because in the original image pixels are closely related. Moreover, it would be better if the quantitative evaluation can still be achieved without the presence of reference image. This is the case of most practical applications. The evaluation metric should provide a measurement of how well the information of the inputs is integrated into the output.

In this paper, the study is limited to the evaluation of pixel-level image fusion. We propose a measurement of fusion performance based on the phase congruency calculation which was suggested by Kovesi [5], [6], [7]. The phase congruency measurement provides an absolute or dimensionless value ranging from 0 to 1; the larger values correspond to the salient features. Two methods are proposed to identify the availability and quality of input features in the fused image. The first one is based on the modified structural similarity measurement, where phase congruency is employed as the structural component. Similarity map with the fused image is generated for each input image. Then, the larger value at each location is retained for overall assessment. The second one is implemented by computing the local cross-correlation of the phase congruency maps between the fused and input images. The index value is obtained by averaging the similarity or cross-correlation value in each pre-defined region. The proposed schemes achieve a no-reference evaluation of the fused image. Experiments are carried out on a group of fused images obtained by various multiresolution fusion algorithms. The validity of the proposed methods are demonstrated.

The rest of the paper is organized as follows. An overview of the metrics used for image fusion is presented in Section 2. In Section 3, the concept and implementation of the feature-based evaluation are described. Experiments with the proposed approach and comparison with other existing methods can be found in Section 4. The experimental results obtained from both the reference-based assessment and blind evaluation are presented. Thus, the proposed algorithms are validated with these tests. Section 5 presents the discussion. In the final Section 6, the conclusion of whole paper is drawn.