Skip to main content
SpringerLink
Log in

TC-net: transformer combined with cnn for image denoising

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In this paper, we propose an effective and novel network architecture based on the transformer TC-Net. The architecture is composed of several transformer blocks and convolutions for image denoising. The following four core designs of TC-Net ensure that it is suitable for image denoising: (1) An extra skip-connection for feature fusion ensures more effective transmission and utilization of low-level and high-level features. (2) Window multihead self-attention greatly reduces the amount of calculation and captures the dependence of capturing long distances. (3) A convolution-based forward network further improves the ability to capture local information. (4) Ingeniously adding a deep residual shrinkage network into a transformer block improves the networks ability to deal with noise and its robustness in complex scenes. Not just for denoising tasks. To deal with similar low-level visual tasks, only the dataset needs to be changed. The model architecture remains the same and is trained separately. Then, the pretraining models of other tasks can be obtained. A large number of experiments proved the ability of TC-Net in image restoration and demonstrated its efficiency and effectiveness in various scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2021) Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp 14821–14831

  2. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2020) Learning enriched features for real image restoration and enhancement. In: Computer vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16, pp 492–511

  3. Tian C, Fei L , Zheng W, Xu Y, Zuo W, Lin C-W (2020) Deep learning on image denoising: an overview. Neural Networks

  4. Tian C, Xu Y, Li Z, Zuo W, Fei L, Liu H (2020) Attention-guided cnn for image denoising. Neural Netw 124:117–129

    Article  Google Scholar 

  5. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv:2010.11929

  6. Yuan K, Guo S, Liu Z, Zhou A, Yu F, Wu W (2021) Incorporating convolution designs into visual transformers. arXiv:2103.11816

  7. Li Y, Zhang K, Cao J, Timofte R, Van Gool L (2021) Localvit: bringing locality to vision transformers. arXiv:2104.05707

  8. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 234–241

  9. Wang Z, Cun X, Bao J, Liu J (2021) Uformer: a general u-shaped transformer for image restoration. arXiv:2106.03106

  10. Zhang Z, Wu C, Coleman S, Kerr D (2020) Dense-inception u-net for medical image segmentation. Comput Methods Prog Biomed 192:105395

    Article  Google Scholar 

  11. Deng X, Dragotti PL (2020) Deep convolutional neural network for multi-modal image restoration and fusion. IEEE Trans Pattern Anal Mach Intell

  12. Mei Y, Fan Y, Zhang Y, Yu J, Zhou Y, Liu D, Fu Y, Huang TS, Shi H (2020) Pyramid attention networks for image restoration. arXiv:2004.13824

  13. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2021) Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14821–14831

  14. Zhao M, Zhong S, Fu X, Tang B, Pecht M (2019) Deep residual shrinkage networks for fault diagnosis. IEEE Trans Industr Inform 16(7):4681–4690

    Article  Google Scholar 

  15. Abdelhamed A, Lin S, Brown MS (2018) A high-quality denoising dataset for smartphone cameras. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1692–1700

  16. Guo S, Yan Z, Zhang K, Zuo W, Zhang L (2019) Toward convolutional blind denoising of real photographs. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1712–1722

  17. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang M-H, Shao L (2020) Cycleisp: real image restoration via improved data synthesis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2696–2705

  18. Cui K, Boev A, Alshina E, Steinbach E (2020) Color image restoration exploiting inter-channel correlation with a 3-stage cnn. IEEE J Sel Top Signal Process 15(2):174–189

    Article  Google Scholar 

  19. Khan A, Sohail A, Zahoora U, Qureshi AS (2020) A survey of the recent architectures of deep convolutional neural networks. Artif Intell Rev 53(8):5455–5516

    Article  Google Scholar 

  20. Lv T, Pan X, Zhu Y, Li L (2021) Unsupervised medical images denoising via graph attention dual adversarial network. Appl Intell 51(6):4094–4105. https://doi.org/10.1007/s10489-020-02016-4

    Article  Google Scholar 

  21. Yang Q, Yan P, Zhang Y, Yu H, Shi Y, Mou X, Kalra MK, Zhang Y, Sun L, Wang G (2018) Low-dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss. IEEE Trans Med Imaging 37(6):1348–1357

    Article  Google Scholar 

  22. Dong Z, Liu G, Ni G, Jerwick J, Duan L, Zhou C (2020) Optical coherence tomography image denoising using a generative adversarial network with speckle modulation. J Biophotonics 13 (4):201960135

    Article  Google Scholar 

  23. Zhang H, Liu J, Yu Z, Wang P (2021) Masg-gan: a multi-view attention superpixel-guided generative adversarial network for efficient and simultaneous histopathology image segmentation and classification. Neurocomputing 463:275–291

    Article  Google Scholar 

  24. Andreini P, Bonechi S, Bianchini M, Mecocci A, Scarselli F (2020) Image generation by gan and style transfer for agar plate image segmentation. Comput Methods Prog Biomed 184:105268

    Article  Google Scholar 

  25. Pan X, Zhan X, Dai B, Lin D, Loy CC, Luo P (2020) Exploiting deep generative prior for versatile image restoration and manipulation. In: European conference on computer vision, pp 262–277

  26. Fu J, Zheng H, Mei T (2017) Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4438–4446

  27. Wang H, Wei M, Cheng R, Yu Y, Zhang X (2021) Residual deep attention mechanism and adaptive reconstruction network for single image super-resolution. Applied Intelligence, https://doi.org/10.1007/s10489-021-02568-z

  28. Zhao M, Zhong S, Fu X, Tang B, Pecht M (2020) Deep residual shrinkage networks for fault diagnosis. IEEE Trans Industr Inform 16(7):4681–4690. https://doi.org/10.1109/TII.2019.2943898

    Article  Google Scholar 

  29. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser U, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  30. Hua W, Liu G (2021) Transformer-based networks over tree structures for code classification. Applied Intelligence, https://doi.org/10.1007/s10489-021-02894-2

  31. Zhao Z, Niu W, Zhang X, Zhang R, Yu Z, Huang C (2021) Trine: syslog anomaly detection with three transformer encoders in one generative adversarial network. Applied Intelligence, https://doi.org/10.1007/s10489-021-02863-9

  32. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European conference on computer vision, pp 213–229

  33. Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training

  34. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv:2010.11929

  35. Liu R, Yuan Z, Liu T, Xiong Z (2021) End-to-end lane shape prediction with transformers. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 3694–3702

  36. Li Y, Zhang K, Cao J, Timofte R, Van Gool L (2021) Localvit: bringing locality to vision transformers. arXiv:2104.05707

  37. Yuan K, Guo S, Liu Z, Zhou A, Yu F, Wu W (2021) Incorporating convolution designs into visual transformers. arXiv:2103.11816

  38. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. arXiv:2103.14030

  39. Xu B, Wang N, Chen T, Li M (2015) Empirical evaluation of rectified activations in convolutional network. arXiv:1505.00853

  40. Shi C, Pun C-M (2019) Adaptive multi-scale deep neural networks with perceptual loss for panchromatic and multispectral images classification. Inf Sci 490:1–17

    Article  MathSciNet  Google Scholar 

  41. Gholizadeh-Ansari M, Alirezaie J, Babyn P (2020) Deep learning for low-dose ct denoising using perceptual loss and edge detection layer. J Digit Imaging 33(2):504–515

    Article  Google Scholar 

  42. Yang Q, Yan P, Zhang Y, Yu H, Shi Y, Mou X, Kalra MK, Zhang Y, Sun L, Wang G (2018) Low-dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss. IEEE Trans Med Imaging 37(6):1348–1357

    Article  Google Scholar 

  43. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision, pp 694–711

  44. Tian C, Xu Y, Li Z, Zuo W, Fei L, Liu H (2020) Attention-guided cnn for image denoising. Neural Netw 124:117–129

    Article  Google Scholar 

  45. Yu S, Park B, Jeong J (2019) Deep iterative down-up cnn for image denoising. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 0–0

  46. Patil TR, Baligar VP, Huilgol RP (2018) Low psnr high fidelity image compression using surrounding pixels. In: International conference on circuits and systems in digital enterprise technology (ICCSDET), pp 1–6

  47. Anwar S, Barnes N (2019) Real image denoising with feature attention. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 3155–3164

  48. Kim Y, Soh JW, Park GY, Cho NI (2020) Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3482–3492

  49. Yue Z, Yong H, Zhao Q, Zhang L, Meng D (2019) Variational denoising network: Toward blind noise modeling and removal. arXiv:1908.11314

  50. Yue Z, Zhao Q, Zhang L, Meng D (2020) Dual adversarial network: toward real-world noise removal and noise generation. In: European conference on computer vision, pp 41–58

  51. Loshchilov I, Hutter F (2017) Decoupled weight decay regularization. arXiv:1711.05101

  52. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  53. Wang H, Xie Q, Zhao Q, Meng D (2020) A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3103–3112

  54. Jiang Y, Chang S, Wang Z (2021) Transgan: two transformers can make one strong gan. arXiv:2102.0707

  55. Kumar V, Choudhary A, Cho E (2020) Data augmentation using pre-trained transformer models. arXiv:2003.02245

  56. Peng X, Wang K, Zhu Z, You Y (2022) Crafting better contrastive views for siamese representation learning. arXiv:2202.03278

Download references

Acknowledgements

This research was supported by the Shaanxi Provincial Technical Innovation Guidance Special (Fund) Plan in 2020 (2020CGXNG-012). We thank Professor Tao Xue for his guidance and mentorship throughout the course of this research.

Author information

Authors and Affiliations

  1. Xi’an Polytechnic University, Xi’an, 710048, China

    Tao Xue & Pengsen Ma

Authors
  1. Tao Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengsen Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tao Xue.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xue, T., Ma, P. TC-net: transformer combined with cnn for image denoising. Appl Intell 53, 6753–6762 (2023). https://doi.org/10.1007/s10489-022-03785-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03785-w

Keywords

Search

Navigation