Abstract
Image restoration (IR) has been extensively studied with lots of excellent strategies accumulated over the years. However, most existing methods still have large room for improvement. In this paper, we boost an unsupervised iterative feature refinement model (IFR) with the enhanced high-dimensional deep mean-shift prior (EDMSP), termed IFR-EDMSP. The proposed model inherits the fantastic noise suppression characteristic of embedded network and the fine detail preservation ability of IFR model. Moreover, based on the fact that multiple implementations of artificial noise in prior learning improve underlying representation capability, three-sigma rule is adopted in IFR-EDMSP for accurate and robust results. Extensive experiments demonstrated that IFR-EDMSP outperforms the typical methods in compressed sensing, image deblurring and super-resolution.









Similar content being viewed by others
References
Engl HW, Kunisch K, Neubauer A (1989) Convergence rates for Tikhonov regularisation of non-linear ill-posed problems[J]. Inverse Prob 5(4):523–540
Rudin LI, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms[J]. Physica D 60(1–4):259–268
Chambolle A (2004) An algorithm for total variation minimization and applications[J]. J Math imaging vision 20(1–2):89–97
Elad M, Aharon M (2006) Image denoising via sparse and redundant representations over learned dictionaries[J]. IEEE Trans Image Process 15(12):3736–3745
Yang J, Wright J, Huang T, et al (2008) Image super-resolution as sparse representation of raw image patches[C]. IEEE conference on computer vision and pattern recognition. IEEE, pp 1–8.
Mairal J, Elad M, Sapiro G (2007) Sparse representation for color image restoration[J]. IEEE Trans Image Process 17(1):53–69
Dong W, Li X, Zhang L, et al (2011) Sparsity-based image denoising via dictionary learning and structural clustering[C]. CVPR 2011. IEEE, pp 457–464.
Qu X, Guo D, Ning B et al (2012) Undersampled MRI reconstruction with patch-based directional wavelets[J]. Magn Reson Imaging 30(7):964–977
Liu Q, Wang S, Ying L et al (2013) Adaptive dictionary learning in sparse gradient domain for image recovery[J]. IEEE Trans Image Process 22(12):4652–4663
Liu Q, Wang S, Yang K et al (2013) Highly undersampled magnetic resonance image reconstruction using two-level Bregman method with dictionary updating[J]. IEEE Trans Med Imaging 32(7):1290–1301
Zhu Y, Zhang Q, Liu Q et al (2015) PANDA-T1ρ: Integrating principal component analysis and dictionary learning for fast T 1 ρ mapping[J]. Magn Reson Med 73(1):263–272
Buades A, Coll B, Morel J M (2005) A non-local algorithm for image denoising[C].2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). IEEE, 2: 60–65.
Dabov K, Foi A, Katkovnik V et al (2007) Image denoising by sparse 3-D transform-domain collaborative filtering[J]. IEEE Trans Image Process 16(8):2080–2095
Liang D, Wang H, Chang Y et al (2011) Sensitivity encoding reconstruction with nonlocal total variation regularization[J]. Magn Reson Med 65(5):1384–1392
Zoran D, Weiss Y (2011) From learning models of natural image patches to whole image restoration[C]. 2011 International Conference on Computer Vision. IEEE, pp 479–486.
Dong W, Zhang L, Shi G et al (2012) Nonlocally centralized sparse representation for image restoration[J]. IEEE Trans Image Process 22(4):1620–1630
Dong W, Shi G, Li X et al (2014) Compressive sensing via nonlocal low-rank regularization[J]. IEEE Trans Image Process 23(8):3618–3632
Gu S, Zhang L, Zuo W, et al (2014) Weighted nuclear norm minimization with application to image denoising[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2862–2869.
Eksioglu EM (2016) Decoupled algorithm for MRI reconstruction using nonlocal block matching model: BM3D-MRI[J]. J Math Imaging Vision 56(3):430–440
Levin A, Fergus R, Durand F, et al (2007) Image and depth from a conventional camera with a coded aperture[J]. ACM transactions on graphics (TOG), 26(3): 70-es.
Bregman LM (1967) The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming[J]. USSR Comput Math Math Phys 7(3):200–217
Huang F, Chen Y, Yin W et al (2010) A rapid and robust numerical algorithm for sensitivity encoding with sparsity constraints: Self-feeding sparse SENSE[J]. Magn Reson Med 64(4):1078–1088
Osher S, Burger M, Goldfarb D et al (2005) An iterative regularization method for total variation-based image restoration[J]. Multiscale Model Simul 4(2):460–489
Wang S, Liu J, Liu Q et al (2016) Iterative feature refinement for accurate undersampled MR image reconstruction[J]. Phys Med Biol 61(9):3291
Cheng J, Jia S, Ying L et al (2018) Improved parallel image reconstruction using feature refinement[J]. Magn Reson Med 80(1):211–223
Burger H C, Schuler C J, Harmeling (2012) Image denoising: Can plain neural networks compete with BM3D?[C]. 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2392–2399.
Chen E (2012) Image denoising and inpainting with deep neural networks[C]. Advances in neural information processing systems. pp 341–349.
Divakar N, Venkatesh Babu R (2017) Image denoising via CNNs: an adversarial approach[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 80–87.
Dong C, Loy C C, He K, et al (2014) Learning a deep convolutional network for image super-resolution[C]. European conference on computer vision. Springer, Cham, pp 184–199.
Wang Z, Liu D, Yang J, et al (2015) Deep networks for image super-resolution with sparse prior[C]. Proceedings of the IEEE international conference on computer vision. pp 370–378.
Dong C, Loy C C, Tang X (2016) Accelerating the super-resolution convolutional neural network[C]. European conference on computer vision. Springer, Cham, pp 391–407.
Kim J, Kwon Lee J, Mu Lee K (2016) Accurate image super-resolution using very deep convolutional networks[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1646–1654.
Dong C, Loy CC, He K et al (2015) Image super-resolution using deep convolutional networks[J]. IEEE Trans Pattern Anal Mach Intell 38(2):295–307
Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3147–3155.
Zhang K, Zuo W, Gu S, et al (2017) Learning deep CNN denoiser prior for image restoration[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3929–3938.
Chen Y, Pock T (2016) Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration[J]. IEEE Trans Pattern Anal Mach Intell 39(6):1256–1272
Tong T, Li G, Liu X, et al (2017) Image super-resolution using dense skip connections[C]. Proceedings of the IEEE International Conference on Computer Vision. pp 4799–4807.
Wang S, Su Z, Ying L, et al (2016) Accelerating magnetic resonance imaging via deep learning[C]. 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). IEEE, pp 514–517.
Sun J, Li H, Xu Z (2016) Deep ADMM-Net for compressive sensing MRI[C]. Advances in neural information processing systems. pp 10–18.
Zhang J, Ghanem B (2018) ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. pp1828–1837.
Zhang Y, Sun L, Yan C et al (2018) Adaptive residual networks for high-quality image restoration[J]. IEEE Trans Image Process 27(7):3150–3163
Liu P, Zhang H, Zhang K, et al (2018) Multi-level wavelet-CNN for image restoration[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 773–782.
Zhang K, Zuo W, Chen Y et al (2017) Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising[J]. IEEE Trans Image Process 26(7):3142–3155
S. A. Bigdeli and M. Zwicker (2018) Image restoration using auto encoding priors[C]. In Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 5: 33–44.
Wang Y, Liu Q, Zhou H et al (2018) Learning multi-denoising autoencoding priors for image super-resolution[J]. J Vis Commun Image Represent 57:152–162
Bigdeli S A, Zwicker M, Favaro P, et al (2017) Deep mean-shift priors for image restoration[C]. Advances in Neural Information Processing Systems. pp 763–772.
Liu R, Ma L, Wang Y et al (2018) Learning converged propagations with deep prior ensemble for image enhancement[J]. IEEE Trans Image Process 28(3):1528–1543
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation[C]. International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015: 234–241.
Agustsson E, Timofte R (2017) Ntire 2017 challenge on single image super-resolution: Dataset and study[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp 126–135.
Liu Q, Leung H (2019) Variable augmented neural network for decolorization and multi-exposure fusion[J]. Information Fusion 46:114–127
Zhu M, Li J, Wang N, et al. Knowledge Distillation for Face Photo-Sketch Synthesis[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Zhu M, Li J, Wang N et al (2019) A deep collaborative framework for face photo–sketch synthesis[J]. IEEE transactions on neural networks and learning systems 30(10):3096–3108
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under 61871206, 61661031. and project of innovative special funds for graduate students YC2019-S052.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhou, J., Meng, M., Xing, J. et al. Iterative feature refinement with network-driven prior for image restoration. Pattern Anal Applic 24, 1623–1634 (2021). https://doi.org/10.1007/s10044-021-01006-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10044-021-01006-7