Xiaolong Liu
ABSTRACT
In the last few years, a myriad of Transformer based methods have drawn considerable attention due to their outstanding performance on various computer vision tasks. However, most image denoising methods are based on convolutional neural networks (CNNs), few attempts have been made with Transformer, especially in self-supervised and unsupervised methods. In this paper, we propose a novel and good performance unsupervised image Denoising Transformer (DnT) which is just trained by the single input noisy image. Our network combines Transformer and CNN to predict the counterpart clean target, the training loss was measured by pairs of noisy independent images constructed from the input image. The dropout-based ensemble is used to get the final denoised result by averaging multiple predictions generated by the trained model. Experiments show that the proposed method not only has superior performance over the state-of-the-art single noisy image denoiser on additive white Gaussian noise (AWGN) removal but also achieves good results on real-world image denoising.
- A. Buades, B. Coll, and J.-M. Morel, “A review of image denoising algorithms, with a new one,” Multiscale Model. Simul., vol. 4, pp. 490–530, 2005.Google Scholar
Cross Ref
- L. P. Yaroslavsky, K. O. Egiazarian, and J. T. Astola, “Transform domain image restoration methods: review, comparison, and interpretation,” in Nonlinear Image Processing and Pattern Analysis XII, vol. 4304. International Society for Optics and Photonics, 2001, pp. 155– 169.Google ScholarCross Ref
- S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 2862–2869.Google ScholarDigital Library
- K. Dabov, A. Foi, V. Katkovnik, and K. O. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” IEEE Transactions on Image Processing, vol. 16, pp. 2080–2095, 2007.Google ScholarCross Ref
- K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” IEEE Transactions on Image Processing, vol. 26, pp. 3142–3155, 2017.Google ScholarDigital Library
- O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in MICCAI, 2015.Google ScholarCross Ref
- Y. Tai, J. Yang, X. Liu, and C. Xu, “Memnet: A persistent memory network for image restoration,” 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4549–4557, 2017.Google ScholarCross Ref
- J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, “Noise2noise: Learning image restoration without clean data,” arXiv preprint arXiv:1803.04189, 2018.Google Scholar
- A. Krull, T.-O. Buchholz, and F. Jug, “Noise2void-learning denoising from single noisy images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2129–Google ScholarCross Ref
- J. Batson and L. Royer, “Noise2self: Blind denoising by self- supervision,” in International Conference on Machine Learning. PMLR, 2019, pp. 524–533.Google Scholar
- D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in Proceedings of the IEEE conference on computer vision and pattern recognition,2018,pp.9446–9454.Google Scholar
- Y. Quan, M. Chen, T. Pang, and H. Ji, “Self2self with dropout: Learning self-supervised denoising from single image,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1887–1895.Google ScholarCross Ref
- A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly , “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.Google Scholar
- T. Pang, H. Zheng, Y. Quan, and H. Ji, “Recorrupted-to-recorrupted: Unsupervised deep learning for image denoising,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2043–2052.Google ScholarCross Ref
- T. Huang, S. Li, X. Jia, H. Lu, and J. Liu, “Neighbor2neighbor: Self- supervised denoising from single noisy images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14 781–14 790.Google ScholarCross Ref
- K. Zhang, W. Zuo, and L. Zhang, “Ffdnet: Toward a fast and flexible solution for cnn-based image denoising,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4608–4622, 2018.Google ScholarCross Ref
- S. Lefkimmiatis, “Universal denoising networks: a novel cnn architecture for image denoising,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3204–3213.Google ScholarCross Ref
- S. Guo, Z. Yan, K. Zhang, W. Zuo, and L. Zhang, “Toward convolutional blind denoising of real photographs,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1712–1722.Google ScholarCross Ref
- Y. Quan, Y. Chen, Y. Shao, H. Teng, Y. Xu, and H. Ji, “Image denoising using complex-valued deep cnn,” Pattern Recognition, vol. 111, p. 107639, 2021.Google ScholarCross Ref
- J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1833–1844.Google ScholarCross Ref
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” arXiv preprint arXiv:2103.14030, 2021.Google Scholar
- S. Laine, T. Karras, J. Lehtinen, and T. Aila, “High-quality self- supervised deep image denoising,” Advances in Neural Information Processing Systems, vol. 32, pp. 6970–6980, 2019.Google Scholar
- X. Wu, M. Liu, Y. Cao, D. Ren, and W. Zuo, “Unpaired learning of deep image denoising,” in European Conference on Computer Vision. Springer, 2020, pp. 352–368.Google ScholarDigital Library
- T. Xiao, P. Dollar, M. Singh, E. Mintun, T. Darrell, and R. Girshick, “Early convolutions help transformers see better,” Advances in Neural Information Processing Systems, vol. 34, 2021.Google Scholar
- Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans- actions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.Google ScholarDigital Library
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014. [Online]. Available: http://jmlr.org/papers/v15/srivastava14a.htmlGoogle ScholarDigital Library
- K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space,” in 2007 IEEE International Conference on Image Processing, vol. 1, 2007, pp. I – 313–I – 316.Google ScholarCross Ref
- J.Xu,H.Li,Z.Liang,D.C.Zhang,andL.Zhang, “Real-world noisy image denoising: A new benchmark,” ArXiv, vol. abs/1804.02603, 2018.Google Scholar
- G. Chen, F. Zhu, and P. A. Heng, “An efficient statistical method for image noise level estimation,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 477–485.Google ScholarDigital Library
- Moran, Nick “Noisier2Noise: Learning to Denoise From Unpaired Noisy Data.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 12061-12069.Google Scholar
- Xu, J., Huang, Y., Cheng, M., Liu, L., Zhu, F., Xu, Z., & Shao, L. (2020). Noisy-as-Clean: Learning Self-Supervised Denoising From Corrupted Image. IEEE Transactions on Image Processing, 29, 9316-9329.Google Scholar
- DnT: Learning Unsupervised Denoising Transformer from Single Noisy Image
Recommendations
Image denoising with complex ridgelets
In this paper, we propose a novel image denoising method by incorporating the dual-tree complex wavelets into the ordinary ridgelet transform. The approximate shift invariant property of the dual-tree complex wavelet and the high directional sensitivity ...
A cross Transformer for image denoising
AbstractDeep convolutional neural networks (CNNs) depend on feedforward and feedback ways to obtain good performance in image denoising. However, how to obtain effective structural information via CNNs to efficiently represent given noisy images is key ...
Highlights- Depth and breadth can search more structural information to improve image denoising performance.
- Achieving multiple interactions can improve adaptability of denoiser for complex scenes.
- Cross Transformer technique extracts more ...
A new wavelet-based fuzzy single and multi-channel image denoising
In this paper, we propose a new wavelet shrinkage algorithm based on fuzzy logic. In particular, intra-scale dependency within wavelet coefficients is modeled using a fuzzy feature. This feature space distinguishes between important coefficients, which ...
Comments