ICWGAN-GP: an image fusion method based on infrared compensator and wasserstein generative adversarial network with gradient penalty

Abstract

The existing Generative adversarial network (GAN)-based infrared (IR) and visible (VIS) image fusion methods mainly used multiple discriminators to preserve salient information in source images, which brings difficulty in balancing the performance of these discriminators during training, leading to unideal fused results. To tackle this disadvantage, an image fusion method based on IR compensator and Wasserstein generative adversarial network with gradient penalty (WGAN-GP) is proposed, called ICWGAN-GP. The generator of ICWGAN-GP employs an adjustment mechanism to obtain more VIS gradients while getting IR intensities, and important details in VIS images are highlighted through the adversarial game between a discriminator and a generator. Using one discriminator allows ICWGAN-GP to focus on learning the feature distribution in a source image, which avoids the balance problem caused by multiple discriminators, and improves the efficiency of the ICWGAN-GP. In addition, an IR compensator based on Quadtree-Bézier method is designed to make up for bright IR features in the fused images. Extensive experiments on public datasets show that ICWGAN-GP can highlight bright target features while generating rich texture in the fused images, and achieves better objective metrics in terms of SCD, CC, FMI_W and VIF than the state-of-the-art methods like U2Fusion, MDLatLRR, DDcGAN, etc. Moreover, in our further fusion tracking experiments, ICWGAN-GP also demonstrates good tracking performance.