Technical SectionCombining attention mechanism and Retinex model to enhance low-light images☆,☆☆
Graphical abstract
Introduction
As digital imaging devices are increasingly widely deployed, people can take photos anytime and anywhere. However, the shooting scenes of many photos are underexposed. Photos taken under underexposure conditions are challenging to show the details of the scenery and people, which cannot meet people’s ideal visual effects and needs. At the same time, these low-visibility photos also bring significant challenges to traditional computer vision tasks such as image segmentation, target detection [1], and tracking [2]. Therefore, designing a practical algorithm to enhance low-light images is necessary.
Although some existing technologies can enhance low-light images, such as setting long exposure, high ISO, and flash, there are some deficiencies with these methods. For example, long exposure has limitations when shooting static scenes, high sensitivity will increase noise and blur the images. High ISO increases the sensitivity of the image sensor to light while also amplifying noise. The utilization of flash can illuminate the environment to a certain extent. Still, it will introduce unexpected highlights and unbalanced light in the photo, making the photo visually unpleasant (see Fig. 1). .
Many researchers have conducted massive research and proposed many solutions to these problems. Early research [3], [4], [5], [6] mainly focused on contrast enhancement. These methods have specific deficiencies in restoring image details and colors. Research in recent years [7], [8], [9], [10] takes deep learning methods to adjust images. These methods simultaneously learn and adjust color, brightness, contrast, and saturation to achieve better results. However, these existing methods still have limitations for enhancing low-light images that are seriously underexposed.
In this paper, we propose the CA&R Net that combines attention mechanism and Retinex model to solve the above mentioned problems. Specifically, we suggest a novel information extraction network that learns to acquire the reflectance (R), illumination (I), and attention map () of the image. Then, is used as a guide for the Restore-Net stage to restore the reflection component. Finally, the recovered reflectivity and low illumination are used to adjust the illumination component. The motivation of our design is that the noise in the underexposed areas of images is more severe than that in the normally exposed areas. Simple enhancement of underexposed areas will amplify the noise hidden in the dark. Therefore, we introduce an attention mechanism to emphasize the vital information of the processed object and suppress some irrelevant details. We extract the attention map to evaluate the degree of underexposure and guide the enhancement in a region-adaptive manner. By doing this, more attention will be paid to the underexposed areas during the enhancement process, which can enhance the underexposed areas and avoid over-enhancing the normal exposure areas. Besides, instead of just using the estimated low illumination for prediction, we use the reconstructed reflectance and low illumination for jointly predicting to illumination layer of the image. This joint prediction utilizes an attention mechanism to make illumination adjustment achieve better results. We conduct extensive experiments on the LOL dataset. Experimental results show that CA&R Net can successfully handle noise, color distortion, and multiple types of degradation with the power of attention information. Furthermore, both SSIM and PSNR outperform Retinex-Net and KinD. Compared with Retinex-Net, CA&R Net dramatically improves the overall performance, with PSNR by 5.38 and SSIM by 26.86%. Compared to KinD, CA&R Net improves PSNR by 1.95 and SSIM by 5.25%.
The main contributions of this paper are summarized as follows:
- •
We propose the CA&R Net that combines attention mechanism and Retinex model to enhance low-light images. With the power of attention information, the CA&R Net can successfully handle noise, color distortion, and multiple types of degradations.
- •
We develop an attention map to guide the reflectance restoration in a region-adaptive manner so that it can pay more attention to underexposed areas during the enhancement process and avoid over-enhancing the normal exposure areas.
- •
Instead of just using the estimated low illumination for prediction, we use the reconstructed reflectance and low illumination to jointly predict the image’s illumination layer. This joint prediction utilizes an attention mechanism to make illumination adjustment achieve better results.
- •
Extensive experiments have been conducted to evaluate our method, and the superiority of our approach has been proved qualitatively and quantitatively.
Section snippets
Related work
Traditional Methods. The earliest low-light image enhancement algorithm is to adjust the light distribution of low-light images uniformly. Histogram equalization (HE) achieves illumination enhancement by expanding the dynamic range of images [11] so that the details hidden in the dark area are redisplayed. Later, some other optimization algorithms were proposed, such as adaptive HE algorithm [12], average intensity retention [13], black and white stretching [14], and a novel LHE algorithm [15].
Retinex model for low-light image enhancement
The classic Retinex theory models human color perception. It assumes that the observed image can be decomposed into two components, reflectance and illumination: where S denotes the input image, R denotes reflectance, I denotes illumination and denotes element-wise multiplication. Reflectance describes the inherent property of the captured object, which is consistent under any brightness conditions. Illumination means various brightness on the object.
For the low-light image enhancement
Implementation details
In this paper, our network is trained using the public dataset LOL, which includes 500 low/normal light image pairs. Among them, 485 image pairs were used for training, 15 image pairs were used for testing, and the synthetic dataset proposed by Fan et al. [44] includes 2458 low/normal light image pairs, of which 2118 pairs are used for training the other 340 pairs for evaluation. Our network consists of three parts: information extraction (including attention and decomposition network),
Conclusion
In this paper, we designed a CA&R Net to solve the problem of low-light image enhancement. We have observed more noise in the underexposed areas of images than in the normally exposed areas. Attention mechanism can be used to emphasize the vital information of the processed object and suppress some irrelevant information. Based on these observations, we propose the CA&R Net that combines attention mechanism and Retinex model to enhance low-light images. The CA&R Net includes three stages of
CRediT authorship contribution statement
Yong Wang: Validation, Formal analysis, Resources, Writing – review & editing, Supervision, Funding acquisition. Jin Chen: Conceptualization, Methodology, Software, Validation, Data curation, Writing – original draft. Yujuan Han: Validation, Formal analysis, Investigation, Data curation, Writing – review & editing, Visualization. Duoqian Miao: Formal analysis, Investigation, Resources, Writing – review & editing, Supervision, Project administration, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (54)
- et al.
A variational-based fusion model for non-uniform illumination image enhancement via contrast optimization and color correction
Signal Process
(2018) - et al.
Image contrast enhancement for preserving mean brightness without losing image features
Eng Appl Artif Intell
(2013) - et al.
A degradation model for simultaneous brightness and sharpness enhancement of low-light image
Signal Process
(2021) - et al.
Single low-light image enhancer using Taylor expansion and fully dynamic convolution
Signal Process
(2021) - et al.
An effective decomposition-enhancement method to restore light field images captured in the dark
Signal Process
(2021) - et al.
LLNet: A deep autoencoder approach to natural low-light image enhancement
Pattern Recognit
(2017) - et al.
A fusion-based enhancing method for weakly illuminated images
Signal Process
(2016) - Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S. Feature pyramid networks for object detection. In:...
- Chu Q, Ouyang W, Li H, Wang X, Liu B, Yu N. Online multi-object tracking using CNN-based single object tracker with...
- et al.
Automatic exposure correction of consumer photographs
Exposure: A white-box photo post-processing framework
ACM Trans Graph
Contrast-limited adaptive histogram equalization: speed and effectiveness
Contrast enhancement system using spatially adaptive histogram equalization with temporal filtering
IEEE Trans Consum Electron
Brightness preserving dynamic histogram equalization for image contrast enhancement
IEEE Trans Consum Electron
A histogram modification framework and its application for image contrast enhancement
IEEE Trans Image Process
Research on the fast algorithm of spatial homomorphic filtering
J Image Grap
Using HSV space real-color image enhanced by homomorphic filter in two channels
Comput Eng Appl
Detail-enhanced exposure fusion
IEEE Trans Image Process
A contrast enhancement method for HDR image using a modified image formation model
IEICE Trans Inf Syst
Fast efficient algorithm for enhancement of low lighting video
The retinex theory of color vision
Sci Am
LIME: Low-light image enhancement via illumination map estimation
IEEE Trans Image Process
Cited by (4)
A Semi-Supervised Network Framework for low-light image enhancement
2023, Engineering Applications of Artificial IntelligenceEditorial Note
2022, Computers and Graphics (Pergamon)Super-Resolution Image Reconstruction Method for Micro Defects of Metal Engine Blades
2023, Guangxue Xuebao/Acta Optica Sinica
- ☆
This work was supported by the National Natural Science Foundation of China (Grant nos. 61502065, and 61976158).
- ☆☆
This article was recommended for publication by J. Zheng.
- 1
Yong Wang and Jin Chen contributed equally to this work, and they are co-first authors.