research-article

CLG-INet: Coupled Local-Global Interactive Network for Image Restoration

Authors:

Jiapeng WangAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 7580 - 7589

https://doi.org/10.1145/3581783.3612251

Published: 27 October 2023 Publication History

Abstract

Image restoration is an ill-posed problem due to the infinite feasible solutions for degraded images. Although CNN-based and Transformer-based approaches have been proven effective in image restoration, there are still two challenges in restoring complex degraded images: 1)local-global information extraction and fusion, and 2)computational cost overhead. To address these challenges, in this paper, we propose a lightweight image restoration network (CLG-INet) based on CNN-Transformer interaction, which can efficiently couple the local and global information. Specifically, our model is hierarchically built with a "sandwich-like" structure of coupling blocks, where each block contains three layers in sequence (CNN-Transformer-CNN). The Transformer layer is designed with two core modules: Dynamic Bi-Projected Attention (DBPA), which performs dual projection with large convolutions across windows to capture long-range dependencies, and Gated Non-linear Feed-Forward Network (GNFF), which reconstructs mixed feature information. In addition, we introduce interactive learning, which fuses local features and global representations in different resolutions to the maximum extent. Extensive experiments demonstrate that CLG-INet significantly boosts performance on various image restoration tasks, such as deraining, deblurring, and denoising.

References

[1]

Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. 2018. A highquality denoising dataset for smartphone cameras. In CVPR. 1692--1700.

[2]

Abdullah Abuolaim and Michael S Brown. 2020. Defocus deblurring using dualpixel data. In ECCV. 111--126.

[3]

Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPR. 126--135.

[4]

Saeed Anwar and Nick Barnes. 2019. Real image denoising with feature attention. In ICCV. 3155--3164.

[5]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv:1607.06450 (2016).

[6]

Yu Bai, Meiqin Liu, Chao Yao, Chunyu Lin, and Yao Zhao. 2023. MSPNet: Multistage progressive network for image denoising. Neurocomputing 517 (2023), 71--80.

Digital Library

[7]

Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, and Donglai Wei. 2021. Learning to generate realistic noisy images via pixel-level noise-aware adversarial training. NeurIPS 34 (2021), 3259--3270.

[8]

Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu. 2020. Spatial-adaptive network for single image denoising. In ECCV 2020. 171--187.

Digital Library

[9]

Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. 2022. Simple baselines for image restoration. In ECCV. 17--33.

[10]

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. 2021. Hinet: Half instance normalization network for image restoration. In CVPR. 182--192.

[11]

Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, and Sung-Jea Ko. 2021. Rethinking coarse-to-fine approach in single image deblurring. In ICCV. 4641--4650.

[12]

Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Color image denoising via sparse 3D collaborative filtering with grouping constraint in luminance-chrominance space. In ICIP 2007, Vol. 1. I--313.

[13]

Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. TIP 16, 8 (2007), 2080--2095.

[14]

Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller. 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In ACM MM. 835--838.

[15]

Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. 2017. Clearing the skies: A deep network architecture for single-image rain removal. TIP 26, 6 (2017), 2944--2956.

Digital Library

[16]

Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017. Removing rain from single images via a deep detail network. In CVPR. 3855--3863.

[17]

Hongyun Gao, Xin Tao, Xiaoyong Shen, and Jiaya Jia. 2019. Dynamic scene deblurring with parameter selective sharing and nested skip connections. In CVPR. 3848--3856.

[18]

Shuhang Gu, Yawei Li, Luc Van Gool, and Radu Timofte. 2019. Self-guided network for fast image denoising. In CVPR. 2511--2520.

[19]

Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2019. Toward convolutional blind denoising of real photographs. In CVPR. 1712--1722.

[20]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.

[21]

Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv:1606.08415 (2016).

[22]

Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In CVPR. 7132--7141.

[23]

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image superresolution from transformed self-exemplars. In CVPR. 5197--5206.

[24]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML. 448--456.

[25]

Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, and Rongrong Ji. 2021. Improving image captioning by leveraging intraand inter-layer global representation in transformer network. In AAAI, Vol. 35. 1655--1663.

[26]

Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. 2020. Multi-scale progressive fusion network for single image deraining. In CVPR. 8346--8355.

[27]

Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiří Matas. 2018. Deblurgan: Blind motion deblurring using conditional adversarial networks. In CVPR. 8183--8192.

[28]

Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In ICCV. 8878-- 8887.

[29]

Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018. Recurrent squeeze-and-excitation context aggregation net for single image deraining. In ECCV. 254--269.

[30]

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV. 10012--10022.

[31]

Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. arXiv:1608.03983 (2016).

[32]

Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv:1711.05101 (2017).

[33]

Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. 2016. Understanding the effective receptive field in deep convolutional neural networks. NeurIPS 29 (2016).

[34]

Yu Luo, Yong Xu, and Hui Ji. 2015. Removing rain from a single image via discriminative sparse coding. In ICCV. 3397--3405.

[35]

Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. NeurIPS 29 (2016).

[36]

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV 2001, Vol. 2. 416--423.

[37]

Tomer Michaeli and Michal Irani. 2013. Nonparametric blind super-resolution. In ICCV. 945--952.

[38]

Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR. 3883--3891.

[39]

Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML. 807--814.

[40]

Dongwon Park, Dong Un Kang, Jisoo Kim, and Se Young Chun. 2020. Multitemporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In ECCV. 327--343.

[41]

Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. 2017. Large kernel matters-improve semantic segmentation by global convolutional network. In CVPR. 4353--4361.

[42]

Tobias Plotz and Stefan Roth. 2017. Benchmarking denoising algorithms with real photographs. In CVPR. 1586--1595.

[43]

Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. 2019. Progressive image deraining networks: A better and simpler baseline. In CVPR. 3937--3946.

[44]

Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. 2020. Real-world blur dataset for learning and benchmarking deblurring algorithms. In ECCV. 184--201.

[45]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI. 234--241.

[46]

Ziyi Shen, Wenguan Wang, Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. 2019. Human-aware motion deblurring. In ICCV. 5572--5581.

[47]

Daqian Shi, Xiaolei Diao, Hao Tang, Xiaomin Li, Hao Xing, and Hao Xu. 2022. RCRN: Real-world Character Image Restoration Network via Skeleton Extraction. In ACM MM. 1177--1185.

[48]

Maitreya Suin, Kuldeep Purohit, and AN Rajagopalan. 2020. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In CVPR. 3606--3615.

[49]

Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. Memnet: A persistent memory network for image restoration. In ICCV. 4539--4547.

[50]

Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. 2018. Scalerecurrent network for deep image deblurring. In CVPR. 8174--8182.

[51]

Radu Timofte, Vincent De Smet, and Luc Van Gool. 2013. Anchored neighborhood regression for fast example-based super-resolution. In ICCV. 1920--1927.

[52]

Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. 2022. Maxim: Multi-axis mlp for image processing. In CVPR. 5769--5780.

[53]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS 30 (2017).

[54]

XiaolongWang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In CVPR. 7794--7803.

[55]

Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. 2022. Uformer: A general u-shaped transformer for image restoration. In CVPR. 17683--17693.

[56]

Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu. 2019. Semisupervised transfer learning for image rain removal. In CVPR. 3877--3886.

[57]

Li Xu, Shicheng Zheng, and Jiaya Jia. 2013. Unnatural l0 sparse representation for natural image deblurring. In CVPR. 1107--1114.

[58]

Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020. Learning texture transformer network for image super-resolution. In CVPR. 5791--5800.

[59]

Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017. Deep joint rain detection and removal from a single image. In CVPR. 1357--1366.

[60]

Wenhan Yang, Robby T Tan, Shiqi Wang, Yuming Fang, and Jiaying Liu. 2020. Single image deraining: From model-based to data-driven and beyond. TPAMI 43, 11 (2020), 4059--4077.

[61]

Rajeev Yasarla and Vishal M Patel. 2019. Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In CVPR. 8405--8414.

[62]

Fisher Yu and Vladlen Koltun. 2015. Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122 (2015).

[63]

Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. 2019. Variational denoising network: Toward blind noise modeling and removal. NeurIPS 32 (2019).

[64]

SyedWaqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient transformer for highresolution image restoration. In CVPR. 5728--5739.

[65]

SyedWaqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2020. Cycleisp: Real image restoration via improved data synthesis. In CVPR. 2696--2705.

[66]

SyedWaqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2020. Learning enriched features for real image restoration and enhancement. In ECCV. Springer, 492--511.

[67]

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-stage progressive image restoration. In CVPR. 14821--14831.

[68]

Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. 2019. Deep stacked hierarchical multi-patch network for image deblurring. In CVPR. 5978--5986.

[69]

Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Selfattention generative adversarial networks. In ICML. 7354--7363.

[70]

He Zhang and Vishal M Patel. 2018. Density-aware single image de-raining using a multi-stream dense network. In CVPR. 695--704.

[71]

He Zhang, Vishwanath Sindagi, and Vishal M Patel. 2019. Image de-raining using a conditional generative adversarial network. TCSVT 30, 11 (2019), 3943--3956.

[72]

Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. 2021. Plug-and-play image restoration with deep denoiser prior. TPAMI 44, 10 (2021), 6360--6376.

Digital Library

[73]

Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li. 2020. Deblurring by realistic blurring. In CVPR. 2737--2746.

[74]

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. TIP 26, 7 (2017), 3142--3155.

Digital Library

[75]

Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In CVPR. 3929--3938.

[76]

Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. TIP 27, 9 (2018), 4608--4622.

[77]

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In CVPR. 2472--2481.

[78]

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2020. Residual dense network for image restoration. TPAMI 43, 7 (2020), 2480--2495.

Cited By

Xie DHu XXiao HZhou YDuan S(2025)Reffusion: Enhancement Conditional Diffusion Framework with Dual Domain Interaction Transformer for image restorationKnowledge-Based Systems10.1016/j.knosys.2025.112998311(112998)Online publication date: Feb-2025
https://doi.org/10.1016/j.knosys.2025.112998
Wu DGuo CLi RMa Z(2024)Cross-Scale Feature Blending Model for Surface Defect Identification in Machine Tool Elements Resilient to Contaminant InterferenceIEEE Access10.1109/ACCESS.2024.350922512(178022-178037)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3509225

Index Terms

CLG-INet: Coupled Local-Global Interactive Network for Image Restoration
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction

Recommendations

Blurred image restoration: A fast method of finding the motion length and angle

Motion blur in photographic images is a result of camera movement or shake. Methods such as Blind Deconvolution are used when information about the direction and size of blur is not known. Restoration methods, such as Lucy and Richardson or Wiener ...
A Blind Image Restoration Method Based on PSF Estimation
WCSE '09: Proceedings of the 2009 WRI World Congress on Software Engineering - Volume 02

In order to improve the quality of the restored image, a blind image restoration method is proposed, by estimating the blur function of the imaging model. Firstly, the parameters of the Gaussian point spread function (PSF) of the observed image are ...
Variational image segmentation model coupled with image restoration achievements

Image segmentation and image restoration are two important topics in image processing with a number of important applications. In this paper, we propose a new multiphase segmentation model by combining image restoration and image segmentation models. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
198
Total Downloads

Downloads (Last 12 months)82
Downloads (Last 6 weeks)6

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xie DHu XXiao HZhou YDuan S(2025)Reffusion: Enhancement Conditional Diffusion Framework with Dual Domain Interaction Transformer for image restorationKnowledge-Based Systems10.1016/j.knosys.2025.112998311(112998)Online publication date: Feb-2025
https://doi.org/10.1016/j.knosys.2025.112998
Wu DGuo CLi RMa Z(2024)Cross-Scale Feature Blending Model for Surface Defect Identification in Machine Tool Elements Resilient to Contaminant InterferenceIEEE Access10.1109/ACCESS.2024.350922512(178022-178037)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3509225

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten