Abstract
To avoid any fallacious assumption on the degeneration procedure in preparing training data, some self-similarity based super-resolution (SR) algorithms have been proposed to exploit the internal recurrence of patches without relying on external datasets. However, the network architectures of those “zero-shot” SR methods are often shallow. Otherwise they would suffer from the over-fitting problem due to the limited samples within a single image. This restricts the strong power of deep neural networks (DNNs). To relieve this problem, we propose a middle-layer feature loss to allow the network architecture to be deeper for handling the video super-resolution (VSR) task in a self-supervised way. Specifically, we constrain the middle-layer feature of VSR network to be as similar as that of the corresponding single image super-resolution (SISR) in a Spatial Module, then fuse the inter-frame information in a Temporal Fusion Module. Experimental results demonstrate that the proposed algorithm achieves significantly superior results on real-world data in comparison with some state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akyildiz, I.F., Ekici, E., Bender, M.D.: MLSR: a novel routing algorithm for multilayered satellite IP networks. IEEE/ACM Trans. Networking 10(3), 411–424 (2002)
Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-gan. arXiv preprint arXiv:1909.06581 (2019)
Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 349–356. IEEE (2009)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)
Huang, J.J., Liu, T., Luigi Dragotti, P., Stathaki, T.: Srhrf+: self-example enhanced single image super-resolution using hierarchical random forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 71–79 (2017)
Huang, Y., Wang, W., Wang, L.: Video super-resolution via bidirectional recurrent convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 1015–1028 (2017)
Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2(2), 109–122 (2016)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lee, S., Choi, M., Lee, K.M.: Dynavsr: Dynamic adaptive blind video super-resolution. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2093–2102 (2021)
Liu, C., Sun, D.: A bayesian approach to adaptive video super resolution. In: CVPR 2011, pp. 209–216. IEEE (2011)
Mittal, A., Moorthy, A.K., Bovik, A.C.: Blind/referenceless image spatial quality evaluator. In: 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), pp. 723–727. IEEE (2011)
Shahar, O., Faktor, A., Irani, M.: Space-time super-resolution from a single video. IEEE (2011)
Shocher, A., Cohen, N., Irani, M.: Zero-shot super-resolution using deep internal learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3118–3126 (2018)
Soh, J.W., Cho, S., Cho, N.I.: Meta-transfer learning for zero-shot super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3516–3525 (2020)
Wu, J., Ma, J., Liang, F., Dong, W., Shi, G., Lin, W.: End-to-end blind image quality prediction with cascaded deep neural network. IEEE Trans. Image Process. 29, 7414–7426 (2020)
Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task-oriented flow. Int. J. Comput. Vis. 127(8), 1106–1125 (2019)
Zontak, M., Irani, M.: Internal statistics of a single natural image. In: CVPR 2011, pp. 977–984. IEEE (2011)
Acknowledgement
This work is partially supported by Guangdong Basic and Applied Basic Reserch Foundation with No. 2021A1515011584 and No.2020A1515110884, and supported by the Education Department of Guangdong Province, PR China, under project No. 2019KZDZX1028. The authors would like to thank the editors and reviewers for their constructive suggestions on our work. The corresponding author of this paper is Fei Zhou.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, C., Luo, H., Liao, G., Lu, Z., Zhou, F., Qiu, G. (2021). Self-Supervised Video Super-Resolution by Spatial Constraint and Temporal Fusion. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13021. Springer, Cham. https://doi.org/10.1007/978-3-030-88010-1_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-88010-1_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88009-5
Online ISBN: 978-3-030-88010-1
eBook Packages: Computer ScienceComputer Science (R0)