A deep learning based image enhancement approach for autonomous driving at night

Abstract

Images of road scenes in low-light situations are lack of details which could increase crash risk of connected autonomous vehicles (CAVs). Therefore, an effective and efficient image enhancement model for low-light images is necessary for safe CAV driving. Though some efforts have been made, image enhancement still cannot be well addressed especially in extremely low light situations (e.g., in rural areas at night without street light). To address this problem, we developed a light enhancement net (LE-net) based on the convolutional neural network. Firstly, we proposed a generation pipeline to transform daytime images to low-light images, and then used them to construct image pairs for model development. Our proposed LE-net was then trained and validated on the generated low-light images. Finally, we examined the effectiveness of our LE-net in real night situations at various low-light levels. Results showed that our LE-net was superior to the compared models, both qualitatively and quantitatively.

Introduction

Benefiting from the rapid development of deep learning approaches in recent years, many computer vision applications have been developed for the design of advanced driver assistance systems (ADASs) and connected autonomous vehicles (CAVs) [1]. These applications mainly focus on object detection [2], [3], object classification [4], [5], object identification [6], [7], semantic segmentation [8], [9], [10], motion estimation [11], and surveillance systems [12]. However, most of the available computer vision applications are based on visible light cameras, and thus can only be used under the condition of normal light and clear weather [13], which makes most of the state-of-the-art models not suitable for nighttime images. Traffic safety statistics show that 51.1% of the U.S. fatal crashes happened in the nighttime (from 6 pm to 6 am), especially in rural areas with extremely low-illumination [14]. Hence, effectively enhancing nighttime images for a clear traffic environment is crucial for traffic safety [15], and should be incorporated into ADASs and CAVs for all-around-the-clock assistance.

However, the outline and appearance details of traffic participants are easily blurred at night, making it difficult to distinguish target objects from the background. Therefore, restoring the details of low-light images is a challenging task, especially for the rural low-light images. Various image enhancement methods have been proposed to solve this problem [16]. One classic method is histogram equalization (HE) [17]. HE makes brightness better distributed on the histogram, which can be used to enhance the local contrast without affecting the overall contrast. Another widely used method is gamma correction [18], [19], which increases the brightness of dark regions by compressing bright pixels. Besides HE and gamma correction, dark channel prior methods [20], retinex-based methods [21], [22], [23] and illumination map estimation methods [24] have also been proposed to address image enhancement problems, among which retinex-based methods are the most popular. More recently, driven by large datasets and the improvement of calculation capabilities, deep learning-based methods (e.g., CNN (convolutional neural network)) have shown great success in image enhancement applications [25], [26], [27], [28]. However, all these approaches only focused on the night situations with external light sources (e.g., urban roads with street light). There is a need for an image enhancement model that can accommodate darker scenarios as accidents are more likely to happen while driving in rural areas at night without street light [14].

To solve this problem and meet the requirements of high precision and real time for practical applications, we proposed a novel method for low-light image generation and an efficient end-to-end pipeline LE-net (light enhancement network) based on CNN to enhance images with different levels of illumination. Specifically, in our proposed low-light image generation method, two commonly used image transformation methods (i.e. gamma correction and contrast scale) were adopted to reduce the illumination of images. For further improvement, histogram matching was used to make the histogram distribution of generated images as close as possible to real night images, and a Gaussian mask based mixup strategy was used for local dimming because not all areas needed to be enhanced in an image. As for image enhancement, previous studies typically used large convolutional networks [25], [26], [27], [28]. A limitation of the applications with large convolutional networks is that they are computationally intensive and cannot meet the real time requirement for practical applications. In our proposed LE-net, a new efficient convolution was introduced to reduce the computation burden, and advanced technologies including linear bottleneck and inversed residual [29], self-attention distillation [30], and feature pyramid module [31] were used to improve the network performance. The main contributions of this study can be summarized as follows.

(1) We developed an effective and efficient CNN based network LE-net to enhance low-light images. The strong nonlinear mapping capability of our LE-net comprehensively models the relationship between our generated image pairs in various illumination situations so that our image enhancement model has a better generalization performance.

(2) We proposed a novel pipeline method for low-light image generation. This method can improve the robustness of our proposed image enhancement model by providing superficially realistic and naturalistic images in various illumination situations for training.

Our efforts can be adopted in the development of ADASs and CAVs to help avoid crashes at night especially in rural areas, which would significantly improve both driving safety for drivers and traffic safety for vulnerable traffic participants.

The remainder of this paper is organized into four sections. Firstly, Section 2 provides a review on the related methods including retinex-based methods and learning-based methods. Section 3 describes our proposed method to generate and enhance low-light images. The corresponding results and the superiority of our proposed models are presented and discussed in Section 4. Finally, Section 5 presents the main conclusions.