Elsevier

Information Fusion

Volume 12, Issue 2, April 2011, Pages 74-84
Information Fusion

Performance comparison of different multi-resolution transforms for image fusion

https://doi.org/10.1016/j.inffus.2010.03.002Get rights and content

Abstract

Image fusion combines information from multiple images of the same scene to get a composite image that is more suitable for human visual perception or further image-processing tasks. In this paper, we compare various multi-resolution decomposition algorithms, especially the latest developed image decomposition methods, such as curvelet and contourlet, for image fusion. The investigations include the effect of decomposition levels and filters on fusion performance. By comparing fusion results, we give the best candidates for multi-focus images, infrared–visible images, and medical images. The experimental results show that the shift-invariant property is of great importance for image fusion. In addition, we also conclude that short filter usually provides better fusion results than long filter, and the appropriate setting for the number of decomposition levels is four.

Introduction

Today, imaging sensors of various types are widely used in military and civilian applications, such as battlefield surveillance, health-care applications, and traffic control. However, the information provided by different imaging sensors may be complementary and redundant. For example, visible image provides the outline of scene, while infrared image can show the existence of some special objects, such as concealed guns or people. To obtain an image that simultaneously contains the outline of scene as well as special objects for the convenience of human visual perception or for further image-processing tasks [1], image fusion can be used to integrate the information provided by individual sensors. In this paper, we concern with the fusion of three types of source images: multi-focus images, infrared–visible images, and medical images.

During the past two decades, many image fusion methods are developed [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12]. According to the stage at which image information is integrated, image fusion algorithms can be categorized into pixel, feature, and decision levels [1]. The pixel-level fusion integrates visual information contained in source images into a single fused image based on the original pixel information [2]. In the past decades, pixel-level image fusion has attracted a great deal of research attention. Generally, these algorithms can be categorized into spatial domain fusion and transform domain fusion [3]. The spatial domain techniques fuse source images using local spatial features, such as gradient, spatial frequency, and local standard derivation [1]. For the transform domain methods, source images are projected onto localized bases which are usually designed to represent the sharpness and edges of an image [3]. Therefore, the transformed coefficients (each corresponds to a transform basis) of an image are meaningful in detecting salient features. Consequently, according to the information provided by transformed coefficients, one can select the required information provided from the source images to construct the fused image.

With the development of different transform bases, many kinds of multi-resolution transforms have been proposed and used for image fusion, including the pyramid decomposition [4], [5], [13], discrete wavelet (DWT) [6], [7], [8], [14], stationary wavelet (SWT) [15], [16], dual-tree complex wavelet (DTCWT) [18], [19], [20], [21], curvelet (CVT) [22], [23], [24], [25], [26], [27], [28], contourlet (CT) [29], [30], [31], and nonsubsampled contourlet transform (NSCT) [32], [33], [34].

Zhang and Blum established a categorization of multiscale decomposition-based image fusion to achieve a high-quality digital camera image [7]. They focused mainly on fusing the multiscale decomposition coefficients. For this reason, only a few basic types were considered, i.e. the Laplacian pyramid transform, the DWT, and the discrete wavelet frame (DWF). Only visible images were considered in performance comparisons for digital camera application. Pajares and Cruz gave a tutorial of the wavelet-based image fusion methods [8]. They presented a comprehensive comparison of different pyramid merging methods, different resolution levels, and different wavelet families. Three fusion examples were provided, namely multi-focus images, multispectral-panchromatic remote sensing images, and functional–anatomical medical images.

Wavelets and related classical multiscale transforms conduct decomposition over a limited dictionary in which the two-dimensional bases simply consist of all possible tensor products of one-dimensional basis functions. To solve this problem, some new multiscale transforms such as curvelet and contourlet are introduced [22], [23], [24], [29]. The main motivation of these transforms is to pursue a “true” two-dimensional transform that can capture the intrinsic geometrical structure [22]. Various transforms have been used to explore the image fusion problem [25], [26], [27], [28], [30], [31], [33], [34]; however, there is a lack of comprehensive comparison of these methods on different types of source images. In addition, notice that the general image fusion framework with pyramid decomposition and wavelet has been well studied [7], [8]. In this paper, we investigate some recently developed multiscale image decomposition methods including the DWT, SWT, DTCWT, CVT, CT, and NSCT, especially different decomposition levels and filters using a general fusion rule.

The rest of this paper is organized as follows. In Section 2, the brief reviews of the DTCWT, CVT, CT, and NSCT are presented. In Section 3, we give the general image fusion framework using multiscale image decomposition. Section 4 presents details of numerical experiments and comprehensive discussions on the results. Finally, the main conclusions of this paper are given in Section 5.

Section snippets

Multi-resolution image decomposition

The multi-resolution transform investigated in this paper includes the DWT, SWT, DTCWT, CVT, CT, and NSCT. The DWT and SWT have been researched extensively, and their principles can be found in many literatures [14], [15]; therefore, in this section, we will only briefly review the DTCWT, CVT, CT, and NSCT.

Generic framework for multiscale-based image fusion

In this paper, we make an assumption that there are just two source images A and B. It should be noticed that the multiscale methods can easily be extended to more source images [7]. Fig. 5 illustrates the generic image fusion framework based on multiscale image decomposition methods. The source images are firstly decomposed into low-frequency subbands and a sequence of high-frequency subbands in different scales and orientations. Then at each position in the transformed subbands, the value

Experimental results and discussion

In this paper, we perform the experiments over 20 pairs of source images, as shown in Fig. 6. These images consist of three types of images, namely multi-focus images (eight pairs shown in Fig. 6a), infrared–visible images (eight pairs shown in Fig. 6b), and medical images (four pairs shown in Fig. 6c). Image registration, which geometrically aligns the source images, should be performed before image fusion. In our study, the source images are assumed to have been registered.

Conclusions

In this paper, we compare the image fusion performance of six multi-resolution transforms with different filters and different numbers of decomposition levels. For each multi-resolution transform, the optimal settings are presented for multi-focus images, infrared–visible images, and medical images, respectively. Then, these optimal settings are compared against each other globally. The experimental results indicate that the appropriate setting for the number of decomposition levels is four. It

Acknowledgements

The authors would like to thank the editor and anonymous reviewers for their detailed review, valuable comments, and constructive suggestions. The authors would also like to thank Professor Arthur Ardeshir Goshtasby (Wright State University) for his insightful suggestions and constructive comments on this work. This paper is supported by the National Natural Science Foundation of China (No. 60871096 and 60835004), the Ph.D. Programs Foundation of Ministry of Education of China (No.

References (39)

  • F. Nencini et al.

    Remote sensing image fusion using the curvelet transform

    Information Fusion

    (2007)
  • L. Yang et al.

    Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform

    Neurocomputing

    (2008)
  • S.Y. Yang et al.

    Image fusion based on a new contourlet packet

    Information Fusion

    (2010)
  • Q. Zhang et al.

    Multi-focus image fusion using the nonsubsampled contourlet transform

    Signal Processing

    (2009)
  • A.A. Goshtasby et al.

    Image fusion: advances in the state of the art

    Information Fusion

    (2007)
  • V.S. Petrovic et al.

    Gradient-based multiresolution image fusion

    IEEE Transactions on Image Processing

    (2004)
  • Z. Zhang et al.

    A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application

    Proceedings of the IEEE

    (1999)
  • P.T. Burt et al.

    The Laplacian pyramid as a compact image code

    IEEE Transactions on Communications

    (1983)
  • S. Mallat

    A Wavelet Tour of Signal Processing

    (2008)
  • Cited by (0)

    View full text