Texture analysis-based multi-focus image fusion using a modified Pulse-Coupled Neural Network (PCNN)

https://doi.org/10.1016/j.image.2020.116068Get rights and content

Highlights

  • An image fusion method based on image decomposition and PCNN is proposed.

  • A modified PCNN model is proposed that is more applicable to multi-focus image processing.

  • Our Method can fuse the color image and gray image.

  • Experiments result shows our method has a good performance.

Abstract

Multi-focus image fusion is an effective method of information fusion that can take a series of source images and obtain a fused image where everything is in focus. In this paper, a multi-focus image fusion method based on image texture that adopts a modified Pulse-Coupled Neural Network (PCNN) approach is proposed. First, the texture of an image is obtained by means of image cartoon and texture decomposition. An ignition image is then acquired by inputting the image textures into a modified PCNN. Ignition images are compared to each other to obtain an initial decision map. A small object detection and bilateral filter is then applied to the initial decision map to reduce noise and enable smoother processing. Finally, the source images and decision map are used to produce the fused image. Experimental results demonstrate that the proposed method effectively preserves the source images information while delivering good image fusion performance.

Introduction

As the development of technology has progressed, so too has research regarding image fusion, driven by a constant quest for ever-better image quality. Image fusion technology can fuse a series of images to obtain a fused image that is clearer than any source image, in ways that are easier for a computer to process than the source image itself. Recently, image fusion research has focused on infrared image fusion, remote sensing image fusion and multi-focus image fusion. Every image fusion technology has its own character, developed to satisfy the demands of different fields of application. Image fusion is primarily used in remoting sensing, biomedicine, intelligence studies and computer vision [1].

Multi-focus image fusion fuses images that have different areas in focus to obtain an ‘all-in-focus’ image [2]. The all-in-focus image is clearer than any source image and contains more information, making it better adapted to both human vision and machine processing. It is also the case that it is easier to analyze an all-in-focus image than a partially-focused image. A partially-focused image will have some parts that are clear others that are blurred.

Partially-focused images are a result of the limitations of the depth of field in a camera and the principles of optical imaging [3]. The depth of field refers to the relatively clear range before and after imaging by imaging equipment. The principles of optical imaging indicate that, when objects are located in the range of the imaging equipment, these objects will appear clear in the image. Otherwise, they will be blurred. As the blurred elements in an image impose limits upon human perception and computer processing, multi-focus image fusion can be used to bring all parts of an image into focus.

The principal methods associated with multi-focus image fusion are spatial field and transformation field [4]. The difference between spatial field and transformation field is that they have a different operational field when the source images are being fused. In spatial field image fusion, the fusion space is an unsigned 8-bit integer (unit 8, range 0–255). The main methods associated with spatial field fusion are Principle Component Analysis (PCA), logic filters and Gray-value weighted averaging. In transformation field image fusion, the source image is transformed to another fusion space, fusing the transformed coefficients, then applying an inverse transformation to the fused coefficients to obtain an all-in-focus image. The main methods associated with transformation field are pyramid decomposition, wavelet decomposition and Non-Subsample Contourlet Transform (NSCT).

As research in this area has progressed, numerous novel fusion methods have been developed, such as deep learning [5], [6], multi-scale decomposition [7], [8], sparse representation [9], image cartoon and texture decomposition [10], [11], and Pulse-Coupled Neural Networks (PCNN) [12], [13], [14]. In [5], the authors provide a deep learning fusion framework for image fusion and assess its advantages and disadvantages. In [6], a deep learning method is proposed, where clear image blocks and blurred image blocks are used to train the deep learning model. In this way, the model can learn the relationship between the source image and a decision map so that a final decision map can be obtained. The fused image is then obtained by processing the decision map and source image.

Recently, multi-scale decomposition methods have become popular. In [7], multi-focus image fusion based on a multi-scale decomposition method is proposed, where the source image is decomposed into two scales to create a mapping between the source image and the parts that are in focus. The robustness of the two-scale decision map can be used to arrive at more accurate boundaries in the focused region, thus providing a more accurate decision map and a better all-in-focus image. In [8], multi-focus image fusion based on multi-scale morphology focus detection is proposed. Here, the morphology is used to obtain the boundary region between the focused and blurred parts of an image. Then, a decision map can be obtained by conducting an evaluation based on the boundary regions and multi-scale decomposition. The final fused image is obtained by applying a weighted fusion rule.

Image cartoon and texture decomposition and sparse representation methods have been widely used in the field of image processing because of their exceptional performance. Cartoon and texture decomposition can be divided in two categories: one is sparse representation [15]; the other is filtering [16]. In [10], a cartoon and texture decomposition method was used for multi-focus image fusion. Here, the cartoon and texture elements were first obtained using the cartoon and texture decomposition method. Then, the cartoon and texture elements were separately fused using a fusion rule to acquire a fused cartoon part and a fused texture part. These two separate parts were then fused using the fusion rule to obtain the final fused image. Sparse representation has some similarities with image decomposition. Sparse representation [9] is first applied to the source image or an image that has already been decomposed. Then, a sparse representation coefficient is used for the fusion. In a final step, the fused image is obtained by restructuring the fused coefficient.

Pulse-Coupled Neural Networks (PCNN) have been widely used for image fusion because they perform especially well in signal processing. A PCNN model used for image fusion is introduced in [12]. The model is used as follows: First of all, pre-processing methods are applied to the source images to obtain processed images. The principal pre-processing method aims to measure clarity. Then, the processed images serve as external stimulus input to a PCNN to obtain ignition images. After this, a decision map is created from the ignition images. Finally, an all-in-focus image is obtained, based on the decision map and source images. In [14], a combined PCNN and random walk-based method is used for multi-focus image fusion. First, a clarity measure is used as external input to the PCNN to obtain ignition images. Then, a decision map is constructed that is based on a comparison of two ignition images. The random walk algorithm used for the decision map can accurately define the boundary of the decision map and modify it accordingly. After this, the fused image is obtained from the source images and modified decision map.

The methods described above have their own advantages and disadvantages. The advantage of multi-focus image fusion based on deep learning [6] is that it delivers an accurate decision map based on training regarding clear image blocks and blurred ones. Its disadvantages [6] are its computational complexity and the interpretative nature of deep learning theory [17], [18], with it requiring numerous training samples at a cost to computer performance. The advantage of multi-focus image fusion based on cartoon and texture decomposition is that the fusion is more detailed because of the consideration of both cartoon and texture elements. Its disadvantage is that the current design of its fusion rule is imperfect and it is unable to save the source image pixels. The advantages of multi-focus methods based on PCNN are that PCNN is computational simple, flexible, and achieves good results. Its disadvantages are that the ignition image is unstable and the boundary between the focused and blurred parts of an image is inaccurate.

A novel multi-focus image fusion method based on texture analysis and PCNN is proposed in this paper that aims to overcome the disadvantages described above. The method works as follows: First, the texture elements in a set of source images are obtained through cartoon and texture decomposition. These texture elements are used as external stimulus input to a PCNN to obtain ignition images. An initial decision map is now derived by comparing the ignition images. Then, an algorithm that can detect small objects is used to remove noise from the pixels in the initial decision map and a bilateral filter is used to smooth the map boundary, thereby producing a final decision map. The fused image is now obtained on the basis of both the final decision map and the source images. This algorithm is able to preserve information about the focus boundary because the texture elements contain information about the parts of an image that were in focus. The pixel information from the source images is thus protected. Experimental results demonstrate that the method performs multi-focus image fusion very effectively. The rest of the paper is organized as follows: In Section 2, PCNN theory is described. In Section 3, our method is described in detail. Section 4 presents the results of an experimental comparison between our method and other image fusion methods. In Section 5, we provide our conclusions.

Section snippets

Related work

In this section, we describe the PCNN model in detail. First of all, we look at the traditional PCNN mathematical model. Then, a modified PCNN model is described that is better-adapted to multi-focus image fusion.

Proposed multi-focus image fusion method

In this section, we will introduce our multi-focus image fusion and describe each step in detail. We will begin with the method’s overall framework, then fully describe every step in the procedure.

Experimental results and analysis

In this section, we report on experiments conducted to verify the effectiveness of the proposed method, together with an analysis of the results. First, the objective evaluation criteria used to assess the method’s viability are presented. Then, our fusion method is compared with other state-of-the-art fusion methods to evaluate its comparative performance.

Conclusion

In this paper, a texture analysis-based method for multi-focus image fusion that uses a modified PCNN has been proposed. As a result of the focused elements in original images being preserved within their most textured regions, a binary image capturing the focused part of an original image can be obtained that can then be analyzed using PCNN. This binary image can be used to fuse the source images and obtain a comprehensively focused image. The main contributions the paper can be described as

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Nature Science Foundation of China (Grant No. 61561010), National Nature Science Foundation in Guangxi, China (Grant No. 2017GXNSFAA198089), Key Research and Development Program Projects in Guangxi, China (Grant No. AB18126003 and AB18221016).

References (46)

  • JiL.P. et al.

    Fingerprint orientation field estimation using ridge projection

    Pattern Recognit.

    (2008)
  • CaoC.G. et al.

    New shape-based auroral oval segmentation driven by LLS-RHT

    Pattern Recognit.

    (2009)
  • WangZ.B. et al.

    Multi-focus image fusion using PCNN

    Pattern Recognit.

    (2010)
  • QiaoT. et al.

    Joint bilateral filtering and spectral similarity-based sparse representation: A generic framework for effective feature extraction and data classification in hyperspectral imaging

    Pattern Recognit.

    (2018)
  • RajpootN. et al.

    A multi-resolution framework for local similarity based image denoising

    Pattern Recognit.

    (2012)
  • LiX. et al.

    Bilateral filter based total variation regularization for sparse hyperspectral image unmixing

    Inf. Sci.

    (2019)
  • ChenH. et al.

    A human perception inspired quality metric for image fusion based on regional information

    Inf. Fusion

    (2007)
  • YangC. et al.

    A novel similarity based quality metric for image fusion

    Inf. Fusion

    (2008)
  • NejatiM. et al.

    Multi-focus image fusion using dictionary-based sparse representation

    Inf. Fusion

    (2015)
  • ZhangY. et al.

    Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure

    Inf. Fusion

    (2017)
  • LiuYu et al.

    Multi-focus image fusion with a deep convolutional neural network

    Inf. Fusion

    (2017)
  • QiuX.H. et al.

    Guided filter-based multi-focus image fusion through focus region detection

    Signal Process., Image Commun.

    (2019)
  • PadmavathiK. et al.

    A novel medical image fusion by combining TV-L1 decomposed textures based on adaptive weighting scheme

    Eng. Sci. Technol. Int. J.

    (2019)
  • Cited by (0)

    View full text