Abstract
In this paper, we adapt shared prior learning for blind single image super-resolution (SISR). From a variational perspective, we are aiming at minimizing an energy functional consisting of a learned data fidelity term and a data-driven prior, where the learnable parameters are computed in a mean-field optimal control problem. In the associated loss functional, we combine a supervised loss evaluated on synthesized observations and an unsupervised Wasserstein loss for real observations, in which local statistics of images with different resolutions are compared. In shared prior learning, only the parameters of the prior are shared among both loss functions. The kernel estimate is updated iteratively after each step of shared prior learning. In numerous numerical experiments, we achieve state-of-the-art results for blind SISR with a low number of learnable parameters and small training sets to account for real applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: CVPR (2017)
Anwar, S., Barnes, N.: Densely residual Laplacian super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2020.3021088
Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-GAN. In: NIPS, pp. 284–293 (2019)
Benamou, J.D., Carlier, G., Cuturi, M., Nenna, L., Peyré, G.: Iterative Bregman projections for regularized transportation problems. SIAM J. Sci. Comput. 37(2), 1111–1138 (2015). https://doi.org/10.1137/141000439
Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010). https://doi.org/10.1137/090769521
Britanak, V., Yip, P.C., Rao, K.: Discrete Cosine and Sine Transforms. Academic Press (2006). https://doi.org/10.1016/b978-012373624-6/50003-5
Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numer. 25, 161–319 (2016)
E, W., Han, J., Li, Q.: A mean-field optimal control formulation of deep learning. Res. Math. Sci. 6(1), 10 (2019). https://doi.org/10.1007/s40687-018-0172-y
Effland, A., Kobler, E., Kunisch, K., Pock, T.: Variational networks: an optimal control approach to early stopping variational methods for image restoration. J. Math. Imaging Vision 62(3), 396–416 (2020). https://doi.org/10.1007/s10851-019-00926-8
Greenspan, H.: Super-resolution in medical imaging. Comput. J. 52(1), 43–63 (2008). https://doi.org/10.1093/comjnl/bxm075
Gu, J., Lu, H., Zuo, W., Dong, C.: Blind super-resolution with iterative kernel correction. In: CVPR (2019)
Hämarik, U., Kaltenbacher, B., Kangro, U., Resmerita, E.: Regularization by discretization in Banach spaces. Inverse Prob. 32(3), 035004 (2016). https://doi.org/10.1088/0266-5611/32/3/035004
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imageNet classification. In: ICCV, pp. 1026–1034 (2015). https://doi.org/10.1109/ICCV.2015.123
Huang, J., Mumford, D.: Statistics of natural images and models. In: CVPR, vol. 1, pp. 541–547 (1999). https://doi.org/10.1109/CVPR.1999.786990
Kingma, D.P., Ba, J.L.: ADAM: a method for stochastic optimization. In: ICLR (2015)
Kobler, E., Effland, A., Kunisch, K., Pock, T.: Total deep variation for linear inverse problems. In: CVPR (2020)
Kobler, E., Effland, A., Kunisch, K., Pock, T.: Total deep variation: a stable regularizer for inverse problems. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3124086
Lee, A.B., Pedersen, K.S., Mumford, D.: The nonlinear statistics of high-contrast patches in natural images. Int. J. Comput. Vis. 54, 83–103 (2003). https://doi.org/10.1023/A:1023705401078
Luo, Z., Huang, Y., Li, S., Wang, L., Tan, T.: Unfolding the alternating optimization for blind super resolution. In: Advances in Neural Information Processing Systems 33 (2020)
Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol. 2, pp. 416–423 (2001)
Mumford, D., Gidas, B.: Stochastic models for generic images. Quart. Appl. Math. 59(1), 85–111 (2001). https://doi.org/10.1090/qam/1811096
Peyré, G., Cuturi, M.: Computational optimal transport: with applications to data science. In: Now Foundations and Trends (2019). https://doi.org/10.1561/2200000073
Pinetz, T., Kobler, E., Pock, T., Effland, A.: Shared prior learning of energy-based models for image reconstruction. SIAM J. Imaging Sci. 14(4), 1706–1748 (2021). https://doi.org/10.1137/20M1380016
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vis. 82(2), 205–229 (2009). https://doi.org/10.1007/s11263-008-0197-6
Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60(1–4), 259–268 (1992). https://doi.org/10.1016/0167-2789(92)90242-F
Santambrogio, F.: Optimal Transport for Applied Mathematicians. PNDETA, vol. 87. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20828-2
Shermeyer, J., Van Etten, A.: The effects of super-resolution on object detection performance in satellite imagery. In: CVPR (2019)
Stockert, J.C., Castro, A.B.: Fluorescence Microscopy In Life Sciences. Bentham Science Publishers (2017)
Sun, W., Chen, Z.: Learned image downscaling for upscaling using content adaptive resampler. IEEE Trans. Image Process. 29, 4027–4040 (2020). https://doi.org/10.1109/TIP.2020.2970248
Villani, C.: Optimal transport: old and new. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-540-71050-9
Xie, Y., Wang, X., Wang, R., Zha, H.: A fast proximal point method for computing exact Wasserstein distance. In: UAI (2019)
Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.H., Liao, Q.: Deep learning for single image super-resolution: a brief review. IEEE Trans. Multimedia 21(12), 3106–3121 (2019). https://doi.org/10.1109/TMM.2019.2919431
Yu, J., Fan, Y., Huang, T.: Wide activation for efficient image and video super-resolution. In: BMVC (2019)
Zhang, K., Van Gool, L., Timofte, R.: Deep unfolding network for image super-resolution. In: CVPR (2020)
Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: CVPR (2017)
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deepresidual channel attention networks. In: ECCV (2018)
Zhang, Z., Wang, Z., Lin, Z., Qi, H.: Image super-resolution by neural texture transfer. In: CVPR (2019)
Zhao, N., Wei, Q., Basarab, A., Dobigeon, N., Kouamé, D., Tourneret, J.Y.: Fast single image super-resolution using a new analytical solution for \(\ell _2-\ell _2\) problems. IEEE Trans. Image Process. 25(8), 3683–3697 (2016)
Zhu, S.C., Wu, Y., Mumford, D.: Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling. Int. J. Comput. Vis. 27(2), 107–126 (1998). https://doi.org/10.1023/A:1007925832420
Funding
This work was supported by the German Research Foundation under Germany’s Excellence Strategy - EXC-2047/1 – 390685813 and – EXC2151 – 390873048.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pinetz, T., Kobler, E., Pock, T., Effland, A. (2022). Blind Single Image Super-Resolution via Iterated Shared Prior Learning. In: Andres, B., Bernard, F., Cremers, D., Frintrop, S., Goldlücke, B., Ihrke, I. (eds) Pattern Recognition. DAGM GCPR 2022. Lecture Notes in Computer Science, vol 13485. Springer, Cham. https://doi.org/10.1007/978-3-031-16788-1_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-16788-1_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16787-4
Online ISBN: 978-3-031-16788-1
eBook Packages: Computer ScienceComputer Science (R0)