Meta-learning based relation and representation learning networks for single-image deraining

Highlights

• We propose the meta-learning based relation and representation learning networks for single-image deraining.

• Our proposed method aims to learn the transferable embeddings of rainy images by characterizing the relation between rainy/clean images.

• Effectiveness of our proposed method is validated through evaluations on different settings by comparing against several state-of-the-art algorithms.

Abstract

Single-image deraining is a kind of computer vision task that aims to restore the image that be degraded by rain streaks, which motivates existing methods to either directly translate the rainy image to its clean one, or indirectly learn the rain residual based on the prior information. However, both methodologies harm the generalization ability due to the limited diversity of the training samples, comparing with the endless varieties of the real-world rainy images. Such fact inspires us to take the merit of meta-learning and propose a meta-learning based representation learning network to learn the transferable embeddings of the rainy/clean images, while their discrepancies are characterized by the relation vector, which is generated by the subsequent meta-learning based relation learning network. These networks are leveraged into the meta-learning based deraining network (MLDN) to enhance the generalization ability by removing the latent relation vector from the transferable embedding of the rainy image and generate high-quality deraining result. Superior performance is achieved by MLDN, which has averaged 4$\%$ better than the state-of-the-arts.

Introduction

Raining is the most common weather condition that affects many outdoor computer vision tasks e.g., video surveillance [1], [2], person-reidentification [3], [4], [5], [6], [7] and object detection [8], [9], [10]. Single-image deraining is a kind of computer vision task that aims to simultaneously remove the rain streaks from the degraded image, while keep the fidelity of the background. Conventional deraining methods usually attempt to remove the implicit feature of the rain streaks from the single rainy image [11], [12]. These algorithms are realized by either modeling the shape of the rain streaks [13] or adopting hand-crafted filters [14]. In addition, deep learning architectures such as convolutional neural network (CNN) [15], [16], long short-term memory (LSTM) [17] and generative adversarial network(GAN) [18] are also employed to map rainy image to its clean one.

In general, the conventional deraining algorithms usually aim to either directly “translate” the entire rainy image that includes both the rain streaks and the clean background to the clean one, or indirectly characterize the rain residual and remove it from the embedding of the rainy image. However, in a departure from other computer vision task e.g., face recognition that has similar sample distributions, the backgrounds of rainy images show endless varieties. Such fact could damage the balance between the removal of the rain streaks and the preservation of the fidelity of the background, when one type of background is far from the data distribution of the training set, which resulting into blurring image. Moreover, the rainy images collected from the real-world don’t have ground-truths, which limits the performances of the conventional deep learning based deraining methods trained on the synthesised datasets.

Recently, meta-learning as a kind of auto machine learning technique that designed for “learning to learn” has been applied to many AI tasks [19], [20]. Its most common implementation is the few-shot learning, where a meta-learning based discriminator is trained by only few samples collected from different categories. These samples can be treated as “metadata” to help the discriminator depict the general distribution of the training data, so that it can rapidly adjust itself to the unseen samples. In this regard, the encoding layers of the discriminator is capable of learning the transferable embeddings of the input samples [21].

Such observation inspires us to learn the transferable embeddings of the rainy/clean images and the rain streaks, which can be rapidly transferred to unseen samples and enhance the deraining network to balance the performance of deraining and generalization. We demonstrate several deraining results of our meta-learning based deraining network (MLDN) in Fig. 1. It can be seen that although the backgrounds of the input rainy images show a big difference, our deraining network can still achieve satisfactory results. More results can be found in Section 4.

To achieve the superior deraining results on real-world rainy images, the major issue is to accurately disentangle and remove the rain streaks. We observe that although the backgrounds are various to each other, the relation between rainy images and clean images is clear: the existence of the similar rain streaks. Such observation motivates us to characterize the discrepancy between rainy/clean images to provide the deraining network with an accurate target to remove. Hence, we first propose a task-tailored meta-learning based relation network to preserve the transferable embeddings of the rain streaks in the relation vector, which is generated by forcing the rainy images with different backgrounds to have higher relevance scores, owing to the existence of rain streaks.

However, due to the endless diversity of the real-world rainy images, a well-characterized relation vector is not enough for the deraining task. Such drawback reveals another crucial problem within the deraining task i.e., the limited diversity of the training samples, which harms the generalization ability of the conventional deraining methods trained on the synthesized dataset. To address this issue, we aim to learn the transferable embeddings of the rainy/clean images based on the previous learned relation vector by proposing a meta-learning based discriminator. The discriminator is first trained to distinguish the rainy/clean images, which can be treated as a 2-way K-shot classification task. In contrast to the conventional meta-learning methods, we seamlessly integrate the relation vector into the discriminator to distinguish the rainy/clean images by evaluating the distances between the images and the feature of rain streaks. Thus, the encoding layers of the discriminator can be treated as the representation learning network, which has to explore the information within the rainy image that is closely related to the rain streaks and simultaneously encode the information within the clean image that is away from the rain streaks. Finally, we propose a meta-learning based deraining network (MLDN) to generate clean image by removing the feature of the rain streaks from the embedding of the rainy image.

As shown in Fig. 2, the architecture of MLDN includes three components. The first component is the representation learning network (Fig. 2(a)), which is part of the meta-learning based discriminator. This component aims to learn the embeddings of the rainy/clean images. The second component is the relation network (Fig. 2(b)), which is the classifier of the meta-learning based discriminator. It aims to characterize the relation between rainy/clean images. The obtained relation vector can be seen as the target that need to be removed. The third component is the deraining network (Fig. 2(c)), which aims to generate clean image by removing the feature of the rain streaks from the embedding of the rainy image.

To sum up, our major contributions are concluded as follows:

• To address the issue that the conventional deep learning based deraining methods are trained on the samples with limited diversity, regardless the endless varieties of the backgrounds, we propose a task-tailored meta-learning based relation network to characterize the general relation between rainy/clean images in the relation vector, which helps to distinguish the rain streaks from different backgrounds.

• To address the issue regarding the lack of ground-truths for the rainy images collected from the real-world, we propose a meta-learning based representation learning network to learn the transferable embeddings of the rainy/clean images and facilitate the ability of generalization.

• We seamlessly integrate the relation network along with the representation learning network into an end-to-end meta-learning based deraining network (MLDN). Our experimental results demonstrate the advantages of our MLDN, especially on real-world rainy images.

The rest of the papers is organized as follows. Section 2 discusses related works. Section 3 presents the MLDN, while Section 4 offers the experimental results and comparative analysis. We conclude the paper in Section 5.