Elsevier

Neurocomputing

Volume 226, 22 February 2017, Pages 182-191
Neurocomputing

A novel infrared and visible image fusion algorithm based on shift-invariant dual-tree complex shearlet transform and sparse representation

https://doi.org/10.1016/j.neucom.2016.11.051Get rights and content

Abstract

In this paper, a novel shift-invariant dual-tree complex shearlet transform (SIDCST) is constructed and applied to infrared and visible image fusion. Firstly, the mathematical morphology is used for the source images. Then, the images are decomposed by SIDCST to obtain the low frequency sub-band coefficients and high frequency sub-band coefficients. For the low frequency sub-band coefficients, a novel sparse representation (SR)-based fusion rule is presented. For the high frequency sub-band coefficients, a scheme based on the theory of adaptive dual-channel pulse coupled neural network (2APCNN) is presented, and the energy of edge is used for the external input of 2APCNN. Finally, the fused image is obtained by performing the inverse SIDCST. The experimental results show that the proposed approach can obtain state-of-the-art performance compared with conventional image fusion methods in terms of both objective evaluation criteria and visual quality.

Introduction

The infrared image records the thermal radiation intensity of the object, reflecting the shape, size and position of the thermal target. And the visible light image is the reflection characteristic of the scene, which contains rich spectrum information and detail information. Fusion of the infrared and visible light image [1] can provide more accurate information about the scene, which makes full use of their complementary. And it has been successfully applied to many areas, such as remote sensing, computer vision, medical imaging, microscopic imaging and military affairs.

During the past years, researchers put forward a great variety of fusion algorithms, which can be roughly classified into two main categories. The first type is spatial domain-based methods [2]. The second type is the transform domain-based methods [3], [4]. The transform domain methods, based on multi-scale transforms, are the most usually used methods. Many multi-scale transforms have been widely applied to image fusion, such as Laplacian pyramid transform, gradient pyramid transform and discrete wavelet transform, etc. But most of the traditional wavelet transform have no directionality, and cannot obtain more detail and edges information. To overcome these disadvantages, researchers put forward other multi-scale geometric analysis tools such as dual tree complex wavelet transform (DTCWT) [5], contourlet transform (CT) [6], non-subsampled contourlet transform (NSCT) [7] and shift-invariant shearlet transform (SIST) [8]. However, due to the sampling operation in the implementation of CT, there is no shift-invariance property and the occurrence of sub-band spectrum aliasing are obvious, which limits its application in image processing field. As a result, the NSCT has been proposed by Cunha et al. It can inherit the best properties of CT and contain the characteristic of shift-invariance, so it has been applied to image fusion and has achieved good fusion effect in NSCT domain [9]. But the computational efficiency of NSCT is low. In 2008, Easley et al. proposed the SIST. It only requires a summation of the shearing filters rather than inverting a directional filter banks, thus the implementation of SIST is more efficient computationally. However, SIST, as well as other traditional wavelet transform, is real wavelet, which cannot effectively describe the detailed information of the source image. And the redundancy of non-subsampled pyramid filter (NSPF) in SIST is 2J (J means the decomposition scale), the more highly the scale increases, the more complex the algorithm is. Naturally, we expect to have such a multi-scale transform that has good directional selectivity, limited redundancy and the characteristic of shift-invariance. Therefore, a new multi-scale transform called shift-invariant dual-tree complex shearlet transform (SIDCST) is proposed by combining DTCWT with shearlet filter (SF) of SIST, and employed in image fusion.

In addition, a good image fusion method not only depends on the wavelet transform but also depends on the fusion rule in transform domain. For the low-pass sub-bands, the general fusion rule is just simply merged by averaging all the source inputs, but the averaging fusion rule causes the loss of some information from the source images. In recent years, sparse representation (SR) [10] based fusion method has become a popular topic in fusion field. In 2013, Yan et al. [11] proposed multi-scale dictionary learning method with wavelet for image denoising. The method takes advantages of the sparse coding framework and wavelet transform, which has obtained good effects in image denoising. Then, in 2015, Liu et al. [12] applied this method to image fusion. However, the fusion rule of maximal l1-norm to select fused sparse vector, which loses easily some useful information of the source image and reduces the contrast of fused image. In order to overcome the problem, a novel sparse representation is proposed to achieve the fusion of low frequency coefficients. The sigmoid function [13] is introduced into SR, and combined with the novel improved sum-modified Laplacian(NSML) of sparse coefficients to adaptively select the sparse vector. For the high frequency sub-band coefficients, conventionally, the absolute value maximum of high-pass coefficients is used as the high-pass fusion, which loses some redundant information between source images easily. PCNN [14] is a novel artificial neural network model and has been widely applied to image fusion [15] because of its characteristics of global coupling and pulse synchronization of neurons. However, it is known that the PCNN has two defects. On the one hand, PCNN lacks automation because the linking strength is set constant. On the other hand, the value of single pixel is used as external inputs, which causes the loss of detailed information. To make up for these defects, a simplified dual-channel adaptive pulse coupled neural network(2APCNN) is proposed to deal with the high frequency coefficients, which can make full use of local image information and effectively extract the details of image, to make an intelligent decision on the selection of high frequency coefficients.

According to aforementioned analysis, to combine the advantages of SR and 2APCNN, a novel fusion algorithm based on the SIDCST framework is proposed in this paper. The low-pass sub-bands are merged with SR-based fusion approach, and the high-pass sub-bands are fused using the 2APCNN rule. Experimental results show that our proposed method outperforms the existing state-of-the-art methods, in terms of visual and quantitative evaluations. The several main contributions of this paper can be summarized as follows:

  • (1)

    The SIDCST is constructed by cascading of DTCWT and SF. It is an excellent tool for multi-scale geometric analysis of images, and has remarkable advantages in image processing over other multi-resolution transform models.

  • (2)

    A novel SR-based fusion rule is proposed by combining the sigmoid function and the NSML of sub-band coefficients to calculate the activity-level of the sparse vector, which preserves more detail information than the absolute-maximum-selecting scheme.

  • (3)

    The 2APCNN is proposed and implemented in high pass sub-bands, which not only inherits the properties of the global coupling and pulse synchronization of the PCNN but also overcomes the above-mentioned problems and the limits of the original model in the image fusion.

The remaining sections of this paper are organized as follows. We first present the theory of SIDCST in Section 2. Section 3 illustrates the relevant theory about SR and 2APCNN. Section 4 describes the main framework of the proposed method, and the exhaustive fusion steps are described. Experimental results and discussion are given in Section 5. Conclusions are presented in the last section.

Section snippets

Shift-invariant dual-tree complex shearlet transform

Although the DTCWT overcomes the shortcomings of the traditional wavelet transform, it has poor directional selectivity for its fixed directions and cannot represent well the detailed information of source image. Guo et al. proposed SIST by cascading of non-subsampled pyramid filter (NSPF) and shearlet filter (SF). However, the redundancy of non-subsampled pyramid transform is 2J, so the computational efficiency of SIST becomes increasingly slow with the increase of scale. The redundancy of

Sparse representation and dictionary learning

In recent years, sparse representation (SR) has been proven an extremely powerful tool in image processing [16], computer vision [17] and object recognition [18]. The SR is that a signal xRn can be approximately represented by a linear combination of a few atoms from an over-complete dictionary DRn×m(n<m) where n is the signal dimension and m is the dictionary size. In other words, signal x can be described as xDα, where αRm is the sparse coefficient with a few non-zero entries. Because the

The framework of SIDCST-based fusion algorithm

For simplicity, only the fusion of two source images is considered while the proposed fusion algorithm can be directly extended to fuse more than two images. Fig. 3. illustrates the schematic diagram of the proposed fusion framework. The general procedure of the proposed algorithm can be summarized as follows:

  • (1)

    The mathematical morphology [20] is used for the source images to obtain the enhanced images.

  • (2)

    The enhanced images are decomposed by the SIDCST to obtain two low-frequency sub-bands and a

Experimental setups

To verify the performance of the proposed fusion algorithm, we test the proposed algorithm through six pairs of the infrared images and visible images(Fig. Fig. 4), which have been registered perfectly. The experiments are done in the PC with the Intel CPU 2.27 GHz and 2G RAM, operating under Matlab R2014a. In order to compare with other methods, seven different fusion methods are adopted for the comparison. These methods are DWT-based [23], CT-based [24], NSCT-PCNN-based [25], SR-based method

Conclusion

In this paper, a new multi-scale geometric analysis tool SIDCST is constructed by cascading DTCWT and SF. It can better represent the source image and capture the detailed information of the source image. Therefore, a novel image fusion algorithm based on SIDCST is proposed in this paper. According to the different characteristics of the low-pass and high-pass sub-bands produced by SIDCST, different fusion strategies are implemented on low-pass and high-pass sub-bands, respectively. The

Acknowledgments

The authors would like to thank the anonymous reviewers and the associate editor for their constructive comments and suggestions to our paper. The paper is jointly supported by the National Natural Science Foundation of China (11172086), the Natural Science Foundation Project of Anhui Province (1308085MA09) and the Science Foundation Project of Education Department of Anhui Province (2013AJZR0039).

Ming Yin received the B.E. degree from Anhui Normal University in 1985. He received M.S. degree and Ph.D. degrees from Hefei University of Technology in 1991 and 2012, respectively. Currently he is a professor in College of Mathematics, Hefei University of Technology. His research interests include wavelet transform, image processing and compressed sensing.

References (28)

  • T. Xiang et al.

    A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain

    Infrared Phys. Technol.

    (2015)
  • C. Yang et al.

    A novel similarity based quality metric for image fusion

    Inf. Fusion

    (2008)
  • W. Li et al.

    An image fusion algorithm based on second generation wavelet transform and its performance evaluation

    Acta Aotomatica Sin.

    (2007)
  • H. Wang

    A new multiwavelet-based approach to image fusion

    Inf. Fusion

    (2007)
  • Cited by (124)

    View all citing articles on Scopus

    Ming Yin received the B.E. degree from Anhui Normal University in 1985. He received M.S. degree and Ph.D. degrees from Hefei University of Technology in 1991 and 2012, respectively. Currently he is a professor in College of Mathematics, Hefei University of Technology. His research interests include wavelet transform, image processing and compressed sensing.

    Puhong Duan was born in Anqing, PR China, in 1991. He received the bachelor degree in School of Mathematics from Suzhou University in 2014. Currently, He is a master student in the College of Mathematics, Hefei University of Technology. His main research interests include multi-scale geometric analysis, image processing and sparse representation.

    Wei Liu was born in Hefei, PR China, in 1987. He received M.S. degree from Hefei University of Technology in 2013. Currently he is a assistant researcher in institute of intelligent machines, Chinese academy sciences. His main research interests include multi-scale geometric analysis, image denoising and image fusion.

    Xiangyu Liang was born in Pingyao, PR China, in 1992. He received the bachelor degree in School of Mathematics from Jiangxi Normal University in 2015. Currently, He is a master student in the College of Mathematics, Hefei University of Technology. His main research interests include wavelet transform and image super resolution reconstruction.

    View full text