Skip to main content
SpringerLink
Log in

Multi-channel deep image prior for image denoising

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Most existing deep learning-based image denoisers heavily rely on supervised learning , where a denoising network is trained with massive noisy–clean training pairs. However, the lack of training pairs severely limits the applicability of this kind of denoisers in practice. Recently, an unsupervised denoiser, the so-called deep image prior (DIP), which can accomplish the task of noise removal only with the given noisy image itself, provides a great flexibility for those scenarios where acquiring a training image set is difficult, and has attracted extensive attention due to its practical significance and broad applications. However, the absence of external information dramatically reduces the upper bound of the denoising performance of the DIP method. In this paper, we propose a simple yet effective multi-channel approximation mechanism to boost the denoising performance of the DIP, where several representative denoisers are exploited to provide additional priors to ensure that the DIP model converges to a more reasonable position in the solution space during the iterative optimization process. Specifically, for a given noisy image, we first use two representative denoisers belonging to the internal and external prior-based denoising methods to denoise it, and the resulting denoised images are called initial denoised images that will be used as target images to constrain the search space. Then, the last layer of the original DIP model is expanded from single channel to multi-channel correspondingly, which allows the DIP model to produce multiple denoised images, called intermediate denoised images, in a single run. Thereby, the total execution time is considerably decreased. Meanwhile, we adopt a novel sharpness-based pseudo-reference image quality metric to automatically stop the iterative optimization process and ensure the image quality of the intermediate denoised images. Finally, the intermediate denoised images with good complementarity are fused to yield the final denoised image with structural patch decomposition-based fusion method. The extensive experiments on synthetic and real-world images show that the proposed method has superior performances over existing non-learning-based and learning-based methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

The data and materials are available from the corresponding author on reasonable request.

References

  1. Yue, Z., Yong, H., Zhao, Q., Zhang, L., Meng, D.: Variational denoising network: Toward blind noise modeling and removal. In: Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada, pp. 1–12 (2019)

  2. Xu, S., Yang, X., Jiang, S.: A fast nonlocally centralized sparse representation algorithm for image denoising. Signal Process. 131(February), 99–112 (2017)

    Article  Google Scholar 

  3. Jo, Y., Chun, S.Y., Choi, J.: Rethinking deep image prior for denoising. In: International Conference on Computer Vision (ICCV 2021), Montreal, QC, Canada, pp. 5087–5096 (2021)

  4. Cheng, Z., Gadelha, M., Maji, S., Sheldon, D.: A Bayesian perspective on the deep image prior. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, USA, pp. 5438–5446 (2019)

  5. Wu, D., Kim, K., Li, Q.: Low-dose CT reconstruction with Noise2Noise network and testing-time fine-tuning. Med. Phys. 48(12), 7657–7672 (2021)

    Article  Google Scholar 

  6. Jaakko, L., Jacob, M., Jon, H., Samuli, L., Tero, K., Miika, A., Timo, A.: Noise2Noise: Learning image restoration without clean data, In: International Conference on Machine Learning (ICML 2018), Stockholm, Sweden, pp. 2971–2980 (2018)

  7. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. Int. J. Comput. Vis. 128, 1867–1888 (2020)

    Article  Google Scholar 

  8. Mataev, G., Milanfar, P., Elad, M.: DeepRED: Deep image prior powered by RED. In: Conference on Computer Vision and Pattern Recognition (CVPR 2019), Long Beach, CA, USA (2019)

  9. Jo, Y., Chun, S.Y., Choi, J.: Rethinking deep image prior for denoising. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, pp. 5067–5076 (2021)

  10. Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)

    Article  MathSciNet  Google Scholar 

  11. Shi, Z., Mettes, P., Maji, S., Snoek, C.G.M.: On measuring and controlling the spectral bias of the deep image prior. Int. J. Comput. Vis. 130(4), 885–908 (2022)

    Article  Google Scholar 

  12. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  13. Yu, L., Luo, J., Xu, S., Chen, X., Xiao, N.: An unsupervised weight map generative network for pixel-level combination of image denoisers. Appl. Sci. 12(12), 6227 (2022)

    Article  Google Scholar 

  14. Huang, H., Lin, L., Tong, R., Hu, H., Zhang, Q., Iwamoto, Y., Han, X., Chen, Y.-W., Wu, J.: Unet 3+: a full-scale connected UNet for medical image segmentation. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Barcelona, Spain, pp. 1055–1059 (2020)

  15. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32th International Conference on International Conference on Machine Learning (ICML), Lille, France, pp. 448–456 (2015)

  16. Nair, V., E.Hinton, G.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML), Haifa, Israel, pp. 807–814 (2010)

  17. Chen, G., Zhu, F., Heng, P.A.: An efficient statistical method for image noise level estimation. In: 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, pp. 477–485 (2015)

  18. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  19. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 2808–2817 (2017)

  20. Zhang, K., Li, Y., Zuo, W., Zhang, L., Gool, L.V., Timofte, R.: Plug-and-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence. in press

  21. Mou, C., Zhang, J., Wu, Z.: Dynamic attentive graph learning for image restoration. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada, pp. 4308–4317 (2021)

  22. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: SwinIR: Image restoration using swin transformer. In: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, Canada, pp. 1833–1844 (2021)

  23. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H.: Restormer: efficient transformer for high-resolution image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, Louisiana, USA, pp. 5728–5739 (2022)

  24. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 416–423 (2001)

  25. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 416–423 (2001)

  26. Guo, X., Li, Y., Ling, H.: LIME: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2017)

  27. Yang, W., Wang, W., Huang, H., Wang, S., Liu, J.: Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Trans. Image Process. 30, 2072–2086 (2021)

  28. Soh, J.W., Cho, S., Cho, N.I.: Meta-transfer learning for zero-shot super-resolution. In: Conference on Computer Vision and Pattern Recognition (CVPR 2020), Seattle, WA, USA, pp. 3513–3522 (2020)

Download references

Acknowledgements

The authors would like to thank the authors of [6, 7, 10, 12, 18,19,20,21,22,23,24] for providing their source code.

Funding

This research was funded by Natural Science Foundation of China, grant number 62162043, and Jiangxi Postgraduate Innovation Special Fund Project, grant number YC2022-s033.

Author information

Authors and Affiliations

  1. School of Mathematics and Computer Sciences, Nanchang University, Nanchang, 330031, China

    Shaoping Xu, Nan Xiao, Changfei Zhou & Minghai Xiong

  2. Affiliated Infectious Disease Hospital, Nanchang University, Nanchang, 330006, China

    Jie Luo

Authors
  1. Shaoping Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nan Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jie Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Changfei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Minghai Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SX contributed to the conception of the study, NX wrote the main manuscript text, JL and CZ contributed significantly to analysis and manuscript preparation, and MX conducted experiments. All authors reviewed the manuscript.

Corresponding author

Correspondence to Shaoping Xu.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, S., Xiao, N., Luo, J. et al. Multi-channel deep image prior for image denoising. SIViP 17, 4395–4404 (2023). https://doi.org/10.1007/s11760-023-02673-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02673-1

Keywords

Search

Navigation