Multi-focus image fusion using HOSVD and edge intensity

https://doi.org/10.1016/j.jvcir.2017.02.006Get rights and content

Highlights

  • The edge intensity metric is proposed.

  • The relationship between the patch energy and the shrink factor of the sigmoid function is constructed.

  • Multi-strategy fusion rule based on edge intensity and patch energy is designed.

  • The proposed fusion method is performed in the HOSVD domain.

Abstract

The purpose of multi-focus image fusion is to integrate the partially focused images into one single image which is focused everywhere. To achieve this purpose, higher order singular value decomposition (HOSVD) and edge intensity (EDI) based multi-focus image fusion method is proposed. The main characteristics of the proposed method includes: 1. an effective and robust sharpness measure based on edge intensity is presented; 2. considering the fact that HOSVD is an effective data-driven decomposition technique and shows the outstanding ability in the representation of high-dimensional data, it is used to decompose multi-focus images; and 3. a multi-strategy fusion rule based on sigmoid function is used to fuse the decomposition coefficients. Furthermore, several experiments are conducted to verify the superiority of the proposed fusion framework in terms of visual and statistical analyses.

Introduction

The process that the multiple images of the same scene are integrated into a single enhanced composite image is named as image fusion. The fused image is more informative and appropriate than any of the individual source images, which will improve visualization and help to further processing such as feature extraction and target detection [1]. One of classical application of image fusion is multi-focus image fusion. Due to the limited depth of field in commonly used optical lenses, the objects at different distances cannot be held all in focus by the imaging devices. Multi-focus image fusion technique can overcome this problem by taking with diverse focuses into a single image.

At present, the existing multi-focus fusion methods can be mainly divided into two groups: spatial domain and transform domain techniques [2]. The first group methods directly fuse source images by a linear combination. In general, they can be classified as: pixel based and region based. The simplest way is to take the average of the source images pixel by pixel. To avoid treating each pixel as the same, a normalized weighted aggregation approach to image fusion is used [3]. However, the above-mentioned methods are often subject to noise and misregistration [4]. To further improve the quality of fused image, some researchers proposed to fuse images by using blocks or segmented region instead of single pixels [4], [5], [6]. It is important that distinguish the sharpness of block or region in image fusion. For instance, A. Goshtasby first discussed registration of multi-focus images and then divided the images into uniform blocks. The selected blocks with high gradient were smoothly blended together to produce the fused image [7], [8]. Li et al. combined the image block based on the spatial frequency (SF) [5]. Miao et al. [9] used energy of image gradients (G) to measure the activity of block. Apart from SF and G, variance and Energy of image gradient (EOG) are also used as sharpness measures. Huang et al. [10] evaluated these performances of sharpness metrics. Although the effect of misregistration or noise on block or region based fusion methods can be reduced, this kind of fusion methods often suffer from block artifacts which obviously degenerate the visual perception of the fused image. Therefore, the multi-resolution transform based fusion methods have become popular in recent years.

The idea is to decompose the source images from space domain into transform domain, and merge the decomposed coefficients using certain fusion rules. Finally, the fused image is reconstructed by the inverse transform. Initially, discrete wavelet transform (DWT) is used in image fusion [10], [11]. For example, reference [11] evaluated the wavelet coefficient distribution by a locally adaptive Laplacian mixture model and a statistical sharpness measure based on this model was proposed and used in multi-focus image fusion. With the development of multi-resolution theory, Li and Wang proposed a multi-focus image fusion method by combining wavelet and curvelet transform [12]. Zhang et al. achieved the fusion process in the nonsubsampled contourlet transform domain (NSCT) [13]. In addition, Peng et al. took Shearlet transform as the multi-resolution analysis tool in image fusion [14]. The above-mentioned researches have recognized that the transform-based fusion methods can improve the contrast of fused image and obtain better signal-to-noise ratio. However, because there is no one-to-one correspondence between pixel values in spatial domain and decomposed coefficients in the multi-scale transform domain, a little change of a single coefficient may cause more changes of some pixel values. The undesirable artifacts may be introduced in the fusion process. To address this problem, an efficient data-driven decomposition technique HOSVD has been used to decompose image [15]. HOSVD is one of most efficient tensor decomposition techniques, which are more beneficial to represent high-dimensional data compared to vector- and matrix-based methods. Moreover, HOSVD does not require to set parameters and thresholds. Therefore, in this paper, a novel multi-focus image fusion method based on HOSVD and edge intensity is proposed. In view of the fact that image fusion relies on local information of source images, the source images are divided into image patches. The corresponding patches are used to construct the sub-tensors and then the decomposition coefficients of them can be obtained by HOSVD. As all know, the key challenge of multi-focus image fusion is how to evaluate the sharpness of patch. To select information from the most informative patch, edge intensity of coefficient based sharpness measure is proposed and used in multi-strategy fusion rule. Meanwhile, the blocking artifacts are eliminated by using a sliding window technique. The proposed multi-focus image fusion method is robust to noise interference and is flexible to combine various fusion strategies.

The paper is organized as follows. The theories of tensor and HOSVD are introduced in Section 2. A novel multi-focus image fusion method based on HOSVD and edge intensity is developed in Section 3. Experimental results are presented in Section 4. Conclusions are drawn in Section 5.

Section snippets

Tensor and HOSVD

Considering the fact that HOSVD is used in the proposed method as an effective decomposition tool, the related conceptions of tensor and HOSVD are introduced firstly. Tensors (multi-way arrays) are generalizations of scalars, vectors and matrices to an arbitrary number of indices [15]. A Nth-order tensor is an element of the tensor product of N vector spaces. A first-order tensor is a vector, a second-order tensor is a matrix and tensors of order three or higher are called higher-order tensors.

Framework and procedure of the proposed method

The whole framework of the proposed multi-focus image fusion method is shown in Fig. 2. The detailed procedure of the proposed algorithm can be summarized as follows.

Step 1: Read the L M×N-dimensional source images B1BL(l=1L) to be fused. It is worthwhile to note that the perfectly registered input images with different focuses are the promise of achieving the multi-focus image fusion. Some researchers discussed image fusion algorithm for unregistered multi-focus images, such as Refs. [7],

Experimental results

To verify the proposed method, experiments are tested on thirteen groups of co-aligned multi-focus images downloaded from www.imagefusion.org and compared with the traditional fusion methods (Gradient Pyramid transform-based fusion method (GP) [20], Filter Subtract Decimate Pyramid transform-based fusion method (FSDP) [21], Discrete Wavelet transform-based fusion method (DWT) [22], Laplacian Pyramid transform-based fusion method (LAP) [21]), the pulse coupled neural networks (PCNN)-based fusion

Conclusion

In this paper, we have developed a novel multi-focus image fusion method based on HOSVD and edge intensity. The HOSVD transform provided better image representation. In addition, the activity level measure of coefficient is evaluated by edge intensity and the attribution of corresponding patches is distinguished by the energy similarity of them. A multi-strategy fusion rule based on sigmoid function is used to fuse the coefficients, where the sigmoid function is defined by the activity level

Acknowledgements

This work was supported by the National Natural Science Foundation of P. R. China under grant nos. 61300151 and 61373055, the Postdoctoral Science Foundation of China under grant nos. 2013M541601 and 1301079C and the Provincial research grant nos. BK20151358 and BK20151202. The Ministry of Housing and Urban-rural Development of the People's Republic of China under grant no. 2015-K8-035. The Fundamental Research Funds for the Central Universities JUSRP51618B and the Equipment Development and

References (28)

  • H. Li et al.

    Multi sensor image fusion using the wavelet transform

    Graph. Models Image Process.

    (1995)
  • X.B. Qu et al.

    Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain

    Acta Autom. Sin.

    (2008)
  • A.A. Goshtasby et al.

    Image fusion: advances in the state of the art

    Inf. Fus.

    (2007)
  • S.T. Li et al.

    Multifocus image fusion using region segmentation and spatial frequency

    Image Vis. Comput.

    (2007)
  • Cited by (0)

    View full text