Haojia Lin
ABSTRACT
Recently, it has been shown that the traditional Image Signal Processing (ISP) can be replaced by deep neural networks due to their superior performance. However, most of these networks require heavy computation burden and thus are far from sufficient to be deployed on resource-limited platforms, including but not limited to mobile devices and FPGA. To tackle this challenge, we propose an automated search framework that derives ISP models with high image quality while satisfying the low-computation requirement. To reduce the search cost, we adopt the weight-sharing strategy by introducing a supernet and decouple the architecture search into two stages, supernet training and hard-aware evolutionary search. With the proposed framework, we can train the ISP model once and quickly find high-performance but low-computation models on multiple devices. Experiments demonstrate that the searched ISP models have an excellent trade-off between image quality and model complexity, i.e., achieve compelling reconstruction quality with more than 90% reduction in FLOPs as compared to the state-of-the-art networks.
Supplemental Material
Available for Download
- Michael Ashikhmin. 2002. A Tone Mapping Algorithm for High Contrast Images. In Proceedings of Eurographics Workshop on Rendering Techniques. 145--156.Google Scholar
- Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T. Barron. 2019. Unprocessing Images for Learned Raw Denoising. In CVPR. 11036--11045.Google Scholar
- Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2020. Once-for-All: Train One Network and Specialize it for Efficient Deployment. In ICLR.Google Scholar
- Han Cai, Ligeng Zhu, and Song Han. 2019. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In ICLR.Google Scholar
- Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. 2018. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 40, 4 (2018), 834--848.Google ScholarCross Ref
- Xiangxiang Chu, Bo Zhang, and Ruijun Xu. 2021. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search. In ICCV. IEEE, 12219--12228.Google Scholar
- Linhui Dai, Xiaohong Liu, Chengqi Li, and Jun Chen. 2020. AWNet: Attentive Wavelet Network for Image ISP. In ECCV Workshops, Vol. 12537. Springer, 185--201.Google Scholar
- Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2018. Neural Architecture Search: A Survey. CoRR, Vol. abs/1808.05377 (2018). showeprint[arXiv]1808.05377Google Scholar
- Michaë l Gharbi, Gaurav Chaurasia, Sylvain Paris, and Fré do Durand. 2016. Deep joint demosaicking and denoising. ACM Trans. Graph., Vol. 35, 6 (2016), 191:1--191:12.Google ScholarDigital Library
- Yuchao Gu, Shang-Hua Gao, Xu-Sheng Cao, Peng Du, Shao-Ping Lu, and Ming-Ming Cheng. 2021. iNAS: Integral NAS for Device-Aware Salient Object Detection. In ICCV. IEEE, 4914--4924.Google Scholar
- Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. 2020. Single Path One-Shot Neural Architecture Search with Uniform Sampling. In ECCV, Vol. 12361. 544--560.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR. IEEE, 770--778.Google Scholar
- Ming-Chun Hsyu, Chih-Wei Liu, Chao-Hung Chen, Chao-Wei Chen, and Wen-Chia Tsai. 2021. CSAnet: High Speed Channel Spatial Attention Network for Mobile ISP. In CVPR Workshops. 2486--2493.Google Scholar
- Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-Excitation Networks. In CVPR. IEEE, 7132--7141.Google Scholar
- Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely Connected Convolutional Networks. In CVPR. IEEE, 2261--2269.Google Scholar
- Andrey Ignatov, Cheng-Ming Chiang, Hsien-Kai Kuo, Anastasia Sycheva, and Radu Timofte. 2021. Learned Smartphone ISP on Mobile NPUs With Deep Learning, Mobile AI 2021 Challenge: Report. In CVPR Workshops. IEEE, 2503--2514.Google Scholar
- Andrey Ignatov, Luc Van Gool, and Radu Timofte. 2020. Replacing Mobile Camera ISP with a Single Deep Learning Model. In CVPR Workshops. IEEE, 2275--2285.Google ScholarCross Ref
- Andrey Ignatov and Radu Timofte. 2020. AIM 2020 Challenge on Learned Image Signal Processing Pipeline. In ECCV Workshops, Vol. 12537. 152--170.Google Scholar
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV, Vol. 9906. Springer, 694--711.Google ScholarCross Ref
- Ngai Ming Kwok, Haiyan Shi, Quang Phuc Ha, Gu Fang, Shengyong Chen, and Xiuping Jia. 2013. Simultaneous Image Color Correction and Enhancement Using Particle Swarm Optimization. Eng. Appl. Artif. Intell., Vol. 26, 10 (2013), 2356--2371.Google ScholarDigital Library
- Hong-Kwai Lam, Oscar C. Au, and Chi-Wah Wong. 2004. Automatic white balancing using luminance component and standard deviation of RGB components [image preprocessing]. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 493--496.Google Scholar
- Xin Li. 2005. Demosaicing by Successive Approximation. IEEE Transactions on Signal Processing, Vol. 14, 3 (2005), 370--379.Google ScholarDigital Library
- Zhetong Liang, Jianrui Cai, Zisheng Cao, and Lei Zhang. 2021. CameraNet: A Two-Stage Framework for Effective Camera ISP Learning. IEEE Trans. Image Process., Vol. 30 (2021), 2248--2262.Google ScholarDigital Library
- Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L. Yuille, and Li Fei-Fei. 2019a. Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation. In CVPR. IEEE, 82--92.Google Scholar
- Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019b. DARTS: Differentiable Architecture Search. In ICLR. 1--13.Google Scholar
- Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. 2018. Multi-Level Wavelet-CNN for Image Restoration. In CVPR Workshops. 773--782.Google Scholar
- Shanto Rahman, Mostafijur Rahman, Mohammad Abdullah-Al-Wadud, Golam Dastegir Al-Quaderi, and Mohammad Shoyaib. 2016. An Adaptive Gamma Correction for Image Enhancement. EURASIP J. Image Video Process., Vol. 2016 (2016), 35.Google ScholarCross Ref
- Sivalogeswaran Ratnasingam. 2019. Deep Camera: A Fully Convolutional Neural Network for Image Signal Processing. In ICCV Workshops. IEEE, 3868--3878.Google Scholar
- Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. 2019. Regularized Evolution for Image Classifier Architecture Search. In AAAI. 4780--4789.Google Scholar
- Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2017. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 39, 6 (2017), 1137--1149.Google ScholarDigital Library
- Eli Schwartz, Raja Giryes, and Alexander M. Bronstein. 2019. DeepISP: Toward Learning an End-to-End Image Processing Pipeline. IEEE Trans. Image Process., Vol. 28, 2 (2019), 912--923.Google ScholarDigital Library
- Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In CVPR. IEEE, 1874--1883.Google Scholar
- Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scale-Recurrent Network for Deep Image Deblurring. In CVPR. IEEE, 8174--8182.Google Scholar
- Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018. ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. In ECCV Workshops, Vol. 11133. Springer, 63--79.Google Scholar
- Z. Wang, E.P. Simoncelli, and A.C. Bovik. 2003. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, Vol. 2. 1398--1402 Vol.2.Google ScholarCross Ref
- H. Xu, L. Yao, Z. Li, X. Liang, and W. Zhang. 2019. Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification. In ICCV. 6648--6657.Google Scholar
- Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. 2020. PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search. In ICLR.Google Scholar
- Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2017. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Trans. Image Process., Vol. 26, 7 (2017), 3142--3155.Google ScholarDigital Library
- Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018a. Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In ECCV, Vol. 11211. Springer, 294--310.Google ScholarCross Ref
- Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018b. Residual Dense Network for Image Super-Resolution. In CVPR. IEEE, 2472--2481.Google Scholar
- Xiawu Zheng, Rongrong Ji, Qiang Wang, Qixiang Ye, Zhenguo Li, Yonghong Tian, and Qi Tian. 2020. Rethinking Performance Estimation in Neural Architecture Search. In CVPR. IEEE, 11353--11362.Google Scholar
- Yu Zhu, Zhenyu Guo, Tian Liang, Xiangyu He, Chenghua Li, Cong Leng, Bo Jiang, Yifan Zhang, and Jian Cheng. 2020. EEDNet: Enhanced Encoder-Decoder Network for AutoISP. In ECCV Workshops, Vol. 12537. 171--184.Google Scholar
- Barret Zoph and Quoc V. Le. 2017. Neural Architecture Search with Reinforcement Learning. In ICLR.Google Scholar
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2018. Learning Transferable Architectures for Scalable Image Recognition. In CVPR. IEEE, 8697--8710.Google Scholar
Index Terms
- Searching Lightweight Neural Network for Image Signal Processing
Recommendations
PyNET-CA: Enhanced PyNET with Channel Attention for End-to-End Mobile Image Signal Processing
Computer Vision – ECCV 2020 WorkshopsAbstractReconstructing RGB image from RAW data obtained with a mobile device is related to a number of image signal processing (ISP) tasks, such as demosaicing, denoising, etc. Deep neural networks have shown promising results over hand-crafted ISP ...
Insightful and Practical Multi-objective Convolutional Neural Network Architecture Search with Evolutionary Algorithms
Advances and Trends in Artificial Intelligence. Artificial Intelligence PracticesAbstractThis paper investigates a comprehensive convolutional neural network (CNN) representation that encodes both layer connections, and computational block attributes for neural architecture search (NAS). We formulate NAS as a bi-objective optimization ...
Fitness landscape analysis of convolutional neural network architectures for image classification
Highlights- Novel application of local optima network analysis to neural architecture landscapes.
AbstractThe global structure of the hyperparameter spaces of neural networks is not well understood and it is therefore not clear which hyperparameter search algorithm will be most effective. In this paper we analyze the landscapes of ...
Comments