Yuqi Jiang
ABSTRACT
Image restoration is an ill-posed problem due to the infinite feasible solutions for degraded images. Although CNN-based and Transformer-based approaches have been proven effective in image restoration, there are still two challenges in restoring complex degraded images: 1)local-global information extraction and fusion, and 2)computational cost overhead. To address these challenges, in this paper, we propose a lightweight image restoration network (CLG-INet) based on CNN-Transformer interaction, which can efficiently couple the local and global information. Specifically, our model is hierarchically built with a "sandwich-like" structure of coupling blocks, where each block contains three layers in sequence (CNN-Transformer-CNN). The Transformer layer is designed with two core modules: Dynamic Bi-Projected Attention (DBPA), which performs dual projection with large convolutions across windows to capture long-range dependencies, and Gated Non-linear Feed-Forward Network (GNFF), which reconstructs mixed feature information. In addition, we introduce interactive learning, which fuses local features and global representations in different resolutions to the maximum extent. Extensive experiments demonstrate that CLG-INet significantly boosts performance on various image restoration tasks, such as deraining, deblurring, and denoising.
- Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. 2018. A highquality denoising dataset for smartphone cameras. In CVPR. 1692--1700.Google Scholar
- Abdullah Abuolaim and Michael S Brown. 2020. Defocus deblurring using dualpixel data. In ECCV. 111--126.Google Scholar
- Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPR. 126--135.Google Scholar
- Saeed Anwar and Nick Barnes. 2019. Real image denoising with feature attention. In ICCV. 3155--3164.Google Scholar
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv:1607.06450 (2016).Google Scholar
- Yu Bai, Meiqin Liu, Chao Yao, Chunyu Lin, and Yao Zhao. 2023. MSPNet: Multistage progressive network for image denoising. Neurocomputing 517 (2023), 71--80.Google ScholarDigital Library
- Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, and Donglai Wei. 2021. Learning to generate realistic noisy images via pixel-level noise-aware adversarial training. NeurIPS 34 (2021), 3259--3270.Google Scholar
- Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu. 2020. Spatial-adaptive network for single image denoising. In ECCV 2020. 171--187.Google ScholarDigital Library
- Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. 2022. Simple baselines for image restoration. In ECCV. 17--33.Google Scholar
- Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. 2021. Hinet: Half instance normalization network for image restoration. In CVPR. 182--192.Google Scholar
- Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. 2021. Rethinking coarse-to-fine approach in single image deblurring. In ICCV. 4641--4650.Google Scholar
- Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space. In ICIP 2007, Vol. 1. I--313.Google ScholarCross Ref
- Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. TIP 16, 8 (2007), 2080--2095.Google ScholarCross Ref
- Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller. 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In ACM MM. 835--838.Google Scholar
- Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. 2017. Clearing the skies: A deep network architecture for single-image rain removal. TIP 26, 6 (2017), 2944--2956.Google ScholarDigital Library
- Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing rain from single images via a deep detail network. In CVPR. 3855--3863.Google Scholar
- Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. 2019. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR. 3848--3856.Google Scholar
- Shuhang Gu, Yawei Li, Luc Van Gool, and Radu Timofte. 2019. Self-guided network for fast image denoising. In CVPR. 2511--2520.Google Scholar
- Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2019. Toward convolutional blind denoising of real photographs. In CVPR. 1712--1722.Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.Google Scholar
- Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv:1606.08415 (2016).Google Scholar
- Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In CVPR. 7132--7141.Google Scholar
- Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image superresolution from transformed self-exemplars. In CVPR. 5197--5206.Google Scholar
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML. 448--456.Google Scholar
- Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, and Rongrong Ji. 2021. Improving image captioning by leveraging intraand inter-layer global representation in transformer network. In AAAI, Vol. 35. 1655--1663.Google ScholarCross Ref
- Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. 2020. Multi-scale progressive fusion network for single image deraining. In CVPR. 8346--8355.Google Scholar
- Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In CVPR. 8183--8192.Google Scholar
- Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In ICCV. 8878-- 8887.Google Scholar
- Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV. 254--269.Google Scholar
- Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV. 10012--10022.Google Scholar
- Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. arXiv:1608.03983 (2016).Google Scholar
- Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv:1711.05101 (2017).Google Scholar
- Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. 2016. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS 29 (2016).Google Scholar
- Yu Luo, Yong Xu, and Hui Ji. 2015. Removing rain from a single image via discriminative sparse coding. In ICCV. 3397--3405.Google Scholar
- Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. NeurIPS 29 (2016).Google Scholar
- David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV 2001, Vol. 2. 416--423.Google ScholarCross Ref
- Tomer Michaeli and Michal Irani. 2013. Nonparametric blind super-resolution. In ICCV. 945--952.Google Scholar
- Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR. 3883--3891.Google Scholar
- Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML. 807--814.Google Scholar
- Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. 2020. Multitemporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In ECCV. 327--343.Google Scholar
- Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. 2017. Large kernel matters-improve semantic segmentation by global convolutional network. In CVPR. 4353--4361.Google Scholar
- Tobias Plotz and Stefan Roth. 2017. Benchmarking denoising algorithms with real photographs. In CVPR. 1586--1595.Google Scholar
- Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. 2019. Progressive image deraining networks: A better and simpler baseline. In CVPR. 3937--3946.Google Scholar
- Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. 2020. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV. 184--201.Google Scholar
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. 234--241.Google Scholar
- Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. 2019. Human-aware motion deblurring. In ICCV. 5572--5581.Google Scholar
- Daqian Shi, Xiaolei Diao, Hao Tang, Xiaomin Li, Hao Xing, and Hao Xu. 2022. RCRN: Real-world Character Image Restoration Network via Skeleton Extraction. In ACM MM. 1177--1185.Google Scholar
- Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. 2020. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In CVPR. 3606--3615.Google Scholar
- Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. Memnet: A persistent memory network for image restoration. In ICCV. 4539--4547.Google Scholar
- Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scalerecurrent network for deep image deblurring. In CVPR. 8174--8182.Google Scholar
- Radu Timofte, Vincent De Smet, and Luc Van Gool. 2013. Anchored neighborhood regression for fast example-based super-resolution. In ICCV. 1920--1927.Google Scholar
- Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. 2022. Maxim: Multi-axis mlp for image processing. In CVPR. 5769--5780.Google Scholar
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS 30 (2017).Google Scholar
- XiaolongWang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In CVPR. 7794--7803.Google Scholar
- Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. 2022. Uformer: A general u-shaped transformer for image restoration. In CVPR. 17683--17693.Google Scholar
- Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu. 2019. Semisupervised transfer learning for image rain removal. In CVPR. 3877--3886.Google Scholar
- Li Xu, Shicheng Zheng, and Jiaya Jia. 2013. Unnatural l0 sparse representation for natural image deblurring. In CVPR. 1107--1114.Google Scholar
- Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020. Learning texture transformer network for image super-resolution. In CVPR. 5791--5800.Google Scholar
- Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep joint rain detection and removal from a single image. In CVPR. 1357--1366.Google Scholar
- Wenhan Yang, Robby T Tan, Shiqi Wang, Yuming Fang, and Jiaying Liu. 2020. Single image deraining: From model-based to data-driven and beyond. TPAMI 43, 11 (2020), 4059--4077.Google ScholarCross Ref
- Rajeev Yasarla and Vishal M Patel. 2019. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR. 8405--8414.Google Scholar
- Fisher Yu and Vladlen Koltun. 2015. Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122 (2015).Google Scholar
- Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. 2019. Variational denoising network: Toward blind noise modeling and removal. NeurIPS 32 (2019).Google Scholar
- SyedWaqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient transformer for highresolution image restoration. In CVPR. 5728--5739.Google Scholar
- SyedWaqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2020. Cycleisp: Real image restoration via improved data synthesis. In CVPR. 2696--2705.Google Scholar
- SyedWaqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2020. Learning enriched features for real image restoration and enhancement. In ECCV. Springer, 492--511.Google Scholar
- Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-stage progressive image restoration. In CVPR. 14821--14831.Google Scholar
- Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. 2019. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR. 5978--5986.Google Scholar
- Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Selfattention generative adversarial networks. In ICML. 7354--7363.Google Scholar
- He Zhang and Vishal M Patel. 2018. Density-aware single image de-raining using a multi-stream dense network. In CVPR. 695--704.Google Scholar
- He Zhang, Vishwanath Sindagi, and Vishal M Patel. 2019. Image de-raining using a conditional generative adversarial network. TCSVT 30, 11 (2019), 3943--3956.Google ScholarCross Ref
- Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. 2021. Plug-and-play image restoration with deep denoiser prior. TPAMI 44, 10 (2021), 6360--6376.Google ScholarDigital Library
- Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. 2020. Deblurring by realistic blurring. In CVPR. 2737--2746.Google Scholar
- Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP 26, 7 (2017), 3142--3155.Google ScholarDigital Library
- Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In CVPR. 3929--3938.Google Scholar
- Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. TIP 27, 9 (2018), 4608--4622.Google ScholarCross Ref
- Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In CVPR. 2472--2481.Google Scholar
- Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2020. Residual dense network for image restoration. TPAMI 43, 7 (2020), 2480--2495.Google ScholarCross Ref
Index Terms
- CLG-INet: Coupled Local-Global Interactive Network for Image Restoration
Recommendations
Blurred image restoration: A fast method of finding the motion length and angle
Motion blur in photographic images is a result of camera movement or shake. Methods such as Blind Deconvolution are used when information about the direction and size of blur is not known. Restoration methods, such as Lucy and Richardson or Wiener ...
A Blind Image Restoration Method Based on PSF Estimation
WCSE '09: Proceedings of the 2009 WRI World Congress on Software Engineering - Volume 02In order to improve the quality of the restored image, a blind image restoration method is proposed, by estimating the blur function of the imaging model. Firstly, the parameters of the Gaussian point spread function (PSF) of the observed image are ...
Variational image segmentation model coupled with image restoration achievements
Image segmentation and image restoration are two important topics in image processing with a number of important applications. In this paper, we propose a new multiphase segmentation model by combining image restoration and image segmentation models. ...
Comments