skip to main content
10.1145/3633637.3633713acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiccprConference Proceedingsconference-collections
research-article

Adaptive Non-Local Regression Prior based on Transformer for Image Deblurring

Published: 28 February 2024 Publication History

Abstract

The non-local regression prior has shown promising results in image deblurring by effectively combining local continuity and non-local self-similarity of images. However, traditional non-local regression priors have limited representation capacity, and existing non-local regression-based methods often require iterative solutions to complex optimization problem. Additionally, these methods involve manually tuning hyper-parameters, which can be difficult and time-consuming. To address these limitations, we propose an adaptive non-local regression prior based on the Transformer model for image deblurring. Different from other learning-based approaches, the self-attention mechanism in the Transformer module better aligns with the modeling process of non-local regression prior, and the global computation capability of Transformer enables effective modeling of global context information. By unfolding the iterative process of the model into a neural network, we introduce a novel deep unfolding network. The proposed network is trained end-to-end, allowing all parameters to be jointly optimized to facilitate image restoration. The extensive experiments show that our proposed model outperforms state-of-the-art model-based and learning-based methods both in terms of PSNR metrics and visual quality.

References

[1]
Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 126–135.
[2]
Antoni Buades, Bartomeu Coll, and J-M Morel. 2005. A non-local algorithm for image denoising. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), Vol. 2. 60–65.
[3]
Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. 1994. Two deterministic half-quadratic regularization algorithms for computed imaging. In Proceedings of 1st international conference on image processing, Vol. 2. 168–172.
[4]
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Transactions on image processing 16, 8 (2007), 2080–2095.
[5]
Aram Danielyan, Vladimir Katkovnik, and Karen Egiazarian. 2011. BM3D frames and variational image deblurring. IEEE Transactions on image processing 21, 4 (2011), 1715–1728.
[6]
Jiangxin Dong, Stefan Roth, and Bernt Schiele. 2020. Deep wiener deconvolution: Wiener meets deep learning for image deblurring. Advances in Neural Information Processing Systems 33 (2020), 1048–1059.
[7]
Weisheng Dong, Peiyao Wang, Wotao Yin, Guangming Shi, Fangfang Wu, and Xiaotong Lu. 2018. Denoising prior driven deep neural network for image restoration. IEEE transactions on pattern analysis and machine intelligence 41, 10 (2018), 2305–2318.
[8]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[9]
Michael Elad and Michal Aharon. 2006. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing 15, 12 (2006), 3736–3745.
[10]
Dong Gong, Zhen Zhang, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, and Yanning Zhang. 2020. Learning deep gradient descent optimization for image deconvolution. IEEE transactions on neural networks and learning systems 31, 12 (2020), 5468–5482.
[11]
Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. 2014. Weighted nuclear norm minimization with application to image denoising. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2862–2869.
[12]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[13]
Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. 2009. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE conference on computer vision and pattern recognition. 1964–1971.
[14]
Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. 2011. Efficient marginal likelihood optimization in blind deconvolution. In CVPR 2011. 2657–2664.
[15]
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. 2021. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2104.05707 (2021).
[16]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision. 1833–1844.
[17]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision. 1833–1844.
[18]
Licheng Liu, CL Philip Chen, Xinge You, Yuan Yan Tang, Yushu Zhang, and Shutao Li. 2017. Mixed noise removal via robust constrained sparse representation. IEEE Transactions on Circuits and Systems for Video Technology 28, 9 (2017), 2177–2189.
[19]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision. 10012–10022.
[20]
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. 2009. Non-local sparse models for image restoration. In 2009 IEEE 12th international conference on computer vision. 2272–2279.
[21]
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. 416–423.
[22]
Max Mignotte. 2008. A non-local regularization strategy for image deconvolution. Pattern recognition letters 29, 16 (2008), 2206–2212.
[23]
Chao Ren, Xiaohai He, and Truong Q Nguyen. 2018. Adjusted non-local regression and directional smoothness for image restoration. IEEE Transactions on Multimedia 21, 3 (2018), 731–745.
[24]
Chao Ren, Xiaohai He, Chuncheng Wang, and Zhibo Zhao. 2021. Adaptive consistency prior based deep network for image denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8596–8606.
[25]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4510–4520.
[26]
Uwe Schmidt, Jeremy Jancsary, Sebastian Nowozin, Stefan Roth, and Carsten Rother. 2015. Cascades of regression tree fields for image restoration. IEEE transactions on pattern analysis and machine intelligence 38, 4 (2015), 677–689.
[27]
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. 2017. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 114–125.
[28]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[29]
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. 2022. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 17683–17693.
[30]
Jingke Yan, Xin Wang, Jingye Cai, Qin Qin, Hao Yang, Qin Wang, Yao Cheng, Tian Gan, Hua Jiang, Jianhua Deng, 2022. Medical image segmentation model based on triple gate MultiLayer perceptron. Scientific Reports 12, 1 (2022), 6103.
[31]
Kai Zhang, Luc Van Gool, and Radu Timofte. 2020. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3217–3226.
[32]
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. 2021. Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 10 (2021), 6360–6376.
[33]
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3929–3938.
[34]
Junmei Zhong and Huifang Sun. 2018. Edge-preserving image denoising based on orthogonal wavelet transform and level sets. Journal of Image and Graphics 6, 2 (2018), 145–151.
[35]
Daniel Zoran and Yair Weiss. 2011. From learning models of natural image patches to whole image restoration. In 2011 international conference on computer vision. 479–486.

Index Terms

  1. Adaptive Non-Local Regression Prior based on Transformer for Image Deblurring

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Other conferences
      ICCPR '23: Proceedings of the 2023 12th International Conference on Computing and Pattern Recognition
      October 2023
      589 pages
      ISBN:9798400707988
      DOI:10.1145/3633637
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 28 February 2024

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Transformer
      2. deep unfolding network
      3. image deblurring
      4. non-local regression prior

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Conference

      ICCPR 2023

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 29
        Total Downloads
      • Downloads (Last 12 months)29
      • Downloads (Last 6 weeks)5
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media