research-article

Windowing Decomposition Convolutional Neural Network for Image Enhancement

Authors:

Chuanjun Zheng,

Yukun LiuAuthors Info & Claims

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

Pages 424 - 432

https://doi.org/10.1145/3474085.3475181

Published: 17 October 2021 Publication History

Abstract

Image enhancement aims to improve the aesthetic quality of images. Most enhancement methods are based on image decomposition techniques. For example, an entire image can be decomposed into a smooth base layer and a residual detail layer. Applying appropriate algorithms to different layers can solve most enhancement problems. Besides decomposing the entire image, the local decomposition approach in local Laplacian filter can also achieve satisfied enhancement results. As a standard convolution is also a local operator that the output values is determined by neighborhood pixels, we observe that the standard convolution can be improved by integrating the local decomposition method for better solving image enhancement problems. Based on this analysis, we propose Windowing Decomposition Convolution (WDC) that decomposes the content of each convolution window by a windowing basic value before applying convolution operation. Using different windowing basic values, the WDC can gather global information and locally separate the processing of different components of images. Moreover, combined with WDC, a new Windowing Decomposition Convolutional Neural Network (WDCNN) is presented. The experimental results show that our WDCNN achieves superior enhancement performance on the MIT-Adobe FiveK and sRGB-SID datasets for noise-free image retouching and low-light noisy image enhancement compared with state-of-the-art techniques.

References

[1]

P. Burt and E. Adelson. 1983. The Laplacian Pyramid as a Compact Image Code. IEEE Transactions on Communications, Vol. 31, 4 (1983), 532--540. https://doi.org/10.1109/TCOM.1983.1095851

[2]

Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. (2011), 97--104.

Digital Library

[3]

C. Chen, Q. Chen, J. Xu, and V. Koltun. 2018a. Learning to See in the Dark. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 3291--3300. https://doi.org/10.1109/CVPR.2018.00347

[4]

J. Chen and and F. Durand S. Paris. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Transactions on Graphics (TOG), Vol. 26, 3 (2007), 103--es.

Digital Library

[5]

Q. Chen, J. Xu, and V. Koltun. 2017. Fast Image Processing with Fully-Convolutional Networks. In 2017 IEEE International Conference on Computer Vision (ICCV). 2516--2525. https://doi.org/10.1109/ICCV.2017.273

[6]

Y. Chen, Y. Wang, M. Kao, and Y. Chuang. 2018b. Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 6306--6314. https://doi.org/10.1109/CVPR.2018.00660

[7]

X. Fu, D. Zeng, Y. Huang, X. Zhang, and X. Ding. 2016. A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2782--2790. https://doi.org/10.1109/CVPR.2016.304

[8]

M. Gharbi, andJ. T. Barron J. Chen, S. W. Hasinoff, and F. Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics (TOG), Vol. 36, 4 (2017), 1--12.

Digital Library

[9]

X. Guo, Y. Li, and H. Ling. 2017. LIME: Low-Light Image Enhancement via Illumination Map Estimation. IEEE Transactions on Image Processing, Vol. 26, 2 (2017), 982--993. https://doi.org/10.1109/TIP.2016.2639450

Digital Library

[10]

Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. 2020. Conditional Sequential Modulation for Efficient Global Image Retouching. In European Conference on Computer Vision. Springer, 679--695.

[11]

K. He, J. Sun, and X. Tang. 2013. Guided Image Filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, 6 (2013), 1397--1409. https://doi.org/10.1109/TPAMI.2012.213

Digital Library

[12]

Y. Hu, H. He, C. Xu, and and S. Lin B. Wang. 2018. Exposure: A White-Box Photo Post-Processing Framework. ACM Trans. Graph., Vol. 37, 2, Article 26 (May 2018), 17 pages. https://doi.org/10.1145/3181974

Digital Library

[13]

Jie Huang, Zhiwei Xiong, Xueyang Fu, Dong Liu, and Zheng-Jun Zha. 2019. Hybrid Image Enhancement With Progressive Laplacian Enhancing Unit. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM '19). Association for Computing Machinery, New York, NY, USA, 1614--1622. https://doi.org/10.1145/3343031.3350855

Digital Library

[14]

J. Huang, P. Zhu, M. Geng, J. Ran, X. Zhou, and P. Wan C. Xing, and X. Ji. 2018. Range scaling global u-net for perceptual image enhancement on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV). 0--0.

[15]

A. Ignatov, N. Kobyshev, R. Timofte, and K. Vanhoey. 2017. DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks. In 2017 IEEE International Conference on Computer Vision (ICCV). 3297--3305. https://doi.org/10.1109/ICCV.2017.355

[16]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125--1134.

[17]

X. Jia, B. B. De, and and L. V. Gool T. Tuytelaars. 2016. Dynamic filter networks. In Advances in neural information processing systems. 667--675.

Digital Library

[18]

D. P. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[19]

Edwin H Land. 1977. The retinex theory of color vision. Scientific american, Vol. 237, 6 (1977), 108--129.

[20]

S. Moran, P. Marza, S. McDonagh, S. Parisot, and G. Slabaugh. 2020. DeepLPF: Deep Local Parametric Filters for Image Enhancement. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12823--12832. https://doi.org/10.1109/CVPR42600.2020.01284

[21]

S. Paris and F. Durand. 2009. A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach. International Journal of Computer Vision, Vol. 81, 1 (2009), 24--52. https://doi.org/10.1007/s11263-007-0110--8

Digital Library

[22]

S. Paris, S. W. Hasinoff, and J. Kautz. 2011. Local laplacian filters: Edge-aware image processing with a laplacian pyramid. ACM Trans. Graph., Vol. 30, 4 (2011), 68.

Digital Library

[23]

J. Park, J. Lee, D. Yoo, and I. S. Kweon. 2018. Distort-and-Recover: Color Enhancement Using Deep Reinforcement Learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5928--5936. https://doi.org/10.1109/CVPR.2018.00621

[24]

Richard R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[25]

H. Su, V. Jampani, D. Sun, O. Gallo, E. Learned-Miller, and J. Kautz. 2019. Pixel-Adaptive Convolutional Neural Networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11158--11167. https://doi.org/10.1109/CVPR.2019.01142

[26]

H. Talebi and P. Milanfar. 2016. Fast Multilayer Laplacian Enhancement. IEEE Transactions on Computational Imaging, Vol. 2, 4 (2016), 496--509. https://doi.org/10.1109/TCI.2016.2607142

[27]

C. Tomasi and R. Manduchi. 1998. Bilateral filtering for gray and color images. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271). IEEE, 839--846.

Digital Library

[28]

R. Wang, Q. Zhang, C. Fu, X. Shen, W. Zheng, and J. Jia. 2019. Underexposed Photo Enhancement Using Deep Illumination Estimation. (2019), 6842--6850. https://doi.org/10.1109/CVPR.2019.00701

[29]

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-Local Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .

[30]

Yang Wang, Yang Cao, Zheng-Jun Zha, Jing Zhang, Zhiwei Xiong, Wei Zhang, and Feng Wu. 2019. Progressive retinex: Mutually reinforced illumination-noise perception network for low-light image enhancement. In Proceedings of the 27th ACM International Conference on Multimedia. 2015--2023.

Digital Library

[31]

K. Xu, X. Yang, B. Yin, and R. W. H. Lau. 2020. Learning to Restore Low-Light Images via Decomposition-and-Enhancement. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2278--2287. https://doi.org/10.1109/CVPR42600.2020.00235

[32]

Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, and Yizhou Yu. 2016. Automatic photo adjustment using deep neural networks. ACM Transactions on Graphics (TOG), Vol. 35, 2 (2016), 1--15.

Digital Library

[33]

Zhenqiang Ying, Ge Li, Yurui Ren, Ronggang Wang, and Wenmin Wang. 2017. A New Low-Light Image Enhancement Algorithm Using Camera Response Model. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops .

[34]

B. Zhang and J. P. Allebach. 2007. Adaptive Bilateral Filter for Sharpness Enhancement and Noise Removal. In 2007 IEEE International Conference on Image Processing, Vol. 4. IV -- 417--IV -- 420. https://doi.org/10.1109/ICIP.2007.4380043

[35]

Q. Zhang, G. Yuan, C. Xiao, L. Zhu, and W.S. Zheng. 2018. High-Quality Exposure Correction of Underexposed Photos. In Proceedings of the 26th ACM International Conference on Multimedia. 582--590.

Digital Library

[36]

W. Zhou, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, Vol. 13, 4 (2004), 600--612.

Digital Library

Cited By

Zheng CZhan YShi LCakmakci OAkşit K(2024)Focal Surface Holographic Light Transport using Learned Spatially Adaptive ConvolutionsSIGGRAPH Asia 2024 Technical Communications10.1145/3681758.3697989(1-4)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3681758.3697989
Wang JWei YZhang ZFan JZhao YYang YWang M(2024)Progressive Stereo Image Dehazing Network via Cross-View Region InteractionIEEE Transactions on Multimedia10.1109/TMM.2024.336891826(7490-7502)Online publication date: 22-Feb-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3368918
Ji ZZheng HZhang ZYe QZhao YXu M(2024)Multi-Scale Interaction Network for Low-Light Stereo Image EnhancementIEEE Transactions on Consumer Electronics10.1109/TCE.2023.328022970:1(3626-3634)Online publication date: Feb-2024
https://doi.org/10.1109/TCE.2023.3280229
Show More Cited By

Index Terms

Windowing Decomposition Convolutional Neural Network for Image Enhancement
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography

Recommendations

Low-light image enhancement based on variational image decomposition
Abstract
Due to the significant differences in brightness regions in real-world images, existing low-light image enhancement methods may lead to insufficient enhancement in low-light regions or over-enhancement in normal-light regions, as well as color ...
Logarithmic Retinex Decomposition-Aided Convolutional Neural Networks for Low-Light Image Enhancement
ICDSP '21: Proceedings of the 2021 5th International Conference on Digital Signal Processing

Maritime accidents kill thousands of lives every year because of the difficulty in rescuing. Usually, the weather conditions at sea are very complicated, especially at midnight. The images captured by the monitor often suffer from low visibility under ...
Pseudo-Retinex decomposition-based unsupervised underwater image enhancement and beyond
Abstract
Underwater images suffer from color casts and low contrast degraded due to wavelength-dependent light scatter and abortion of the underwater environment. To effectively improve the quality of the underwater images, deep learning-based ...
Highlights
- The proposed UUIE only needs distortion-free terrestrial images in the training process, which reduces the requirement on the paired dataset.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '21: Proceedings of the 29th ACM International Conference on Multimedia

October 2021

5796 pages

ISBN:9781450386517

DOI:10.1145/3474085

General Chairs:
Heng Tao Shen
University of Electronic Science&Technology of China, China
,
Yueting Zhuang
Zhejiang University, China
,
John R. Smith
IBM, USA
,
Program Chairs:
Yang Yang
University of Electronic Science and Technology of China, China
,
Pablo Cesar
CWI&TU Delft, The Netherlands
,
Florian Metze
FACEBOOK, Inc., USA
,
Balakrishnan Prabhakaran
University of Texas at Dallas, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Natural Science Foundation China (NSFC)
Ministry of Science and Technology China (MOST)
Shenzhen Science and Technology Innovation Commission (SZSTI)

Conference

MM '21

Sponsor:

SIGMM

MM '21: ACM Multimedia Conference

October 20 - 24, 2021

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
323
Total Downloads

Downloads (Last 12 months)53
Downloads (Last 6 weeks)10

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zheng CZhan YShi LCakmakci OAkşit K(2024)Focal Surface Holographic Light Transport using Learned Spatially Adaptive ConvolutionsSIGGRAPH Asia 2024 Technical Communications10.1145/3681758.3697989(1-4)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3681758.3697989
Wang JWei YZhang ZFan JZhao YYang YWang M(2024)Progressive Stereo Image Dehazing Network via Cross-View Region InteractionIEEE Transactions on Multimedia10.1109/TMM.2024.336891826(7490-7502)Online publication date: 22-Feb-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3368918
Ji ZZheng HZhang ZYe QZhao YXu M(2024)Multi-Scale Interaction Network for Low-Light Stereo Image EnhancementIEEE Transactions on Consumer Electronics10.1109/TCE.2023.328022970:1(3626-3634)Online publication date: Feb-2024
https://doi.org/10.1109/TCE.2023.3280229
Zheng HZhang ZFan JHong RYang YYan SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Decoupled Cross-Scale Cross-View Interaction for Stereo Image Enhancement in the DarkProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611962(1475-1484)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611962
Kosugi SYamasaki T(2023)Crowd-Powered Photo Enhancement Featuring an Active Learning Based Local FilterIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.323398933:7(3145-3158)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1109/TCSVT.2023.3233989
Wang JZhao SZhang ZZhao YZhang H(2023)Physical-Property Guided End-to-End Interactive Image Dehazing NetworkInternational Conference on Neural Computing for Advanced Applications10.1007/978-981-99-5847-4_9(116-131)Online publication date: 30-Aug-2023
https://doi.org/10.1007/978-981-99-5847-4_9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten