BE-ACGAN: Photo-realistic residual bit-depth enhancement by advanced conditional GAN

Highlights

• An advanced conditional generative adversarial network is proposed for BE task.

• The BE-ACGAN is trained to reconstruct residual images rather than HBD ones.

• The discriminator is improved by adopting ZP images as additional inputs.

• A more reliable loss function is involved to stabilize the adversarial training.

Abstract

Since more demands for high quality visualization have been raised in various fields, monitors with higher bit-depth (HBD) become popular in recent years. However, most digital images are at low bit-depth (LBD) and usually of low visual quality with annoying false contours when displayed on HBD monitors directly. To reconstruct visually pleasant HBD images, many bit-depth enhancement (BE) algorithms have been proposed from various aspects, but the recovered HBD images are usually unsatisfactory with conspicuous false contours or over-blurred textures. Inspired by discriminative learning, we propose a residual BE algorithm based on advanced conditional generative adversarial network (BE-ACGAN), in which the discriminator adversarially helps assess image quality and train the generator to achieve more photo-realistic recovery performance. Besides, since it is hard to distinguish between the reconstructed and real HBD images with similar structures, the discriminator takes residual images as input and further takes LBD images as conditions to achieve more reliable performance. In addition, we present a novel loss function to deal with the difficulty of unstable adversarial training. The proposed algorithm outperforms the state-of-the-art methods on large-scale benchmark datasets. Source codes are available at https://github.com/TJUMMG/BE- ACGAN/.

Introduction

Using more colors to display allows finer color graduations, smoother gradients and more details, which can improve the visual quality significantly. Therefore, a number of studies on wide color gamut (WCG) and high dynamic range (HDR) [1] have been conducted for vivid and realistic displays. 10-bit (i.e., 1,024 colors) and 16-bit (i.e., 65,536 colors) monitors [2] have been increasingly used for high quality visualization. However, most mainstream images and digital image acquisition equipments are of 8-bit or lower bit-depth. Besides, some image lossy compress [3] operations also decrease the content bit-depth. These LBD images suffer from false contour artifacts and chroma distortions when linearly de-quantized to HBD ones and displayed on high dynamic range (HDR) screens. Note that bit-depth enhancement is the inverse problem of quantization, which aims to reconstruct the missing least significant bits. It is quite different from inverse tone mapping [4], which tries to recover the lost details due to non-linear tone mapping [5] operators.

Numerous BE algorithms have been proposed from various aspects, and they have achieved relative good performance. Pixel-wise algorithms, including Zero Padding (ZP) and Bit Replication (BR) [6]. ZP is the basis of the BE algorithm, which directly fills the missing least significant bits (LSBs) [7] with zero. BR is slightly improved on the basis of ZP, which enhances the overall brightness of the image. Though they are of high efficiency, the reconstructed HBD images are visually uncomfortable with annoying false contour artifacts since the surrounding structural features are ignored. To eliminate these false contour artifacts, plenty of context-aware algorithms are proposed. Content Adaptive Image Bit-depth Expansion (CA) [8] and Contour Region Reconstruction (CRR) [9] reconstruct HBD pixel values by neighborhood-flooding based on the distances from the surrounding false contours. They can greatly eliminate false contours, but the details in local minimum/maximum regions are usually over-blurred. In addition, Maximum a Posteriori Estimation of AC Signal (ACDC) [10] and Intensity Potential for Adaptive De-quantization (IPAD) [11] are proposed to reconstruct HBD images from the aspect of graph signal processing and natural image statistics. Although they outperforms other unsupervised algorithms, the false contours are not entirely eliminated. Recently, supervised BE algorithms based on deep learning have been proposed. Bit-Depth Enhancement via Convolutional Neural Network (BE-CNN) [12] is a simple end-to-end network, which takes advantage of gradually expanded receptive field. Bit-depth Enhancement by Concatenating All Level Features of Deep Neural Network (BE-CALF) [13] introduced skip connections between every two layers to perserve structural features. Besides, it is discovered that the residual images are easier to reconstruct than the HBD images. Additionally, Deep Bit-Depth Expansion Network (BDEN) [14] proposed a two-stream network to construct flat and non-flat areas separately, which further suppressed false contours in flat areas. Effective CNN frameworks are carefully designed to better reconstruct HBD images, however, the HBD images recovered by these methods still loss some details and are far away from photo-realistic.

Recently, GAN [15] and its variants [16], [17], [18] have been widely adopted for many computer vision tasks to generate photo-realistic images. Many training procedures [19], [20], [21] are also proposed since it is challenging to find a Nash equilibrium in adversarial training. Moreover, GAN generally outperforms simple generative networks for various image enhancement tasks, such as super-resolution [22], [23], [24], [25], and there is no reason for the BE task to be an exception.

Since BE is a specific image enhancement task in bit-depth of pixel value, the expected finer quantized HBD images can be simply obtained by pixel-wise adding the coarser quantized LBD images and corresponding residual images. Moreover, the residual images with pixel values limited to a quantization step are easier to reconstruct/distinguish than HBD ones. Therefore, in this paper, we propose to reconstruct HBD images indirectly by reconstructing the residual between HBD images and LBD ones with BE-ACGAN, in which the generator learns to reconstruct residual images, and the discriminator is trained to evaluate the residual image recovery performance and help train the generator adversarially. Besides, to improve the distinguish performance and further enhance the reliability of the generator, LBD images are used as conditional inputs of the discriminator. We also propose a more reliable adversarial loss function for the BE task, which can greatly stabilize the adversarial training. Extensive experiments on large-scale datasets show superior recovery performance of BE-ACGAN both objectively and subjectively. The main contributions of BE-ACGAN are summarized as follows:

• We explore the superior performance of GAN for BE task and present a novel algorithm known as BE-ACGAN, which involves residual learning approach.

• In order to enhance the discriminative performance, LBD images with useful quantization information are fed into the discriminator as conditions.

• We compare the proposed BE-ACGAN from other GAN-based image enhancement algorithms, and found that the GAN-based algorithm is more suitable for BE task with the proposed residual learning approach.

The rest of the paper is organized as follows. We thoroughly describe the proposed algorithm in Section 2. Then the experimental results and analysis are given in Section 3. Section 4 compares the difference between the BE-ACGAN and other GAN-based image enhancement methods. Finally, the conclusion of the paper is provided in Section 5.