Abstract
Increasing the resolution of digital images and videos using digital super-resolution (SR) techniques has been of great interest in industry and academia over the past three decades. Most SR methods target improving only the spatial resolution of images and videos, whereas improving the temporal resolution could be more critical for some videos. Motion blur is a temporal artifact by nature, so removing it using spatial SR techniques would be highly challenging and often unsuccessful. This paper proposes a multi-frame motion-based video super-resolution method to increase both spatial and temporal resolutions of a single input video. Our optimization problem is based on a maximum a posteriori estimator that estimates each high-resolution (HR) frame by fusing multiple low-resolution frames. The form of the image prior used in the optimization framework is based on the assumption that natural HR frames are piecewise smooth. We introduce a new method to enhance the sharpness of edges in the video frames during the optimization process. We also involve a temporal constraint that improves temporal consistency in the estimated video. Moreover, we propose a new scheme for motion estimation that better suits video frame rate upsampling. Our results are compared with state-of-the-art SR methods, including ML-based ones, which confirm the effectiveness of the proposed method.
Similar content being viewed by others
Data availability
No dataset is used by this research.
Notes
\(\left\lfloor \cdot \right\rfloor\) is the floor operator.
A few first and last frames of the video may have less number of adjacent frames. Also, for real-time applications, \(b\) should be set to zero.
Generative Adversarial Networks.
References
Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. Inst. Electr. Electron. Eng. Inc 20(3), 21–36 (2003). https://doi.org/10.1109/MSP.2003.1203207
Tian, J., Ma, K.K.: A survey on super-resolution imaging. Signal Image Video Process 5(3), 329–342 (2011). https://doi.org/10.1007/s11760-010-0204-6
Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Signal Process. 128, 389–408 (2016). https://doi.org/10.1016/j.sigpro.2016.05.002
Liu, C., Sun, D.: A Bayesian approach to adaptive video super resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 209–216. (2011). https://doi.org/10.1109/CVPR.2011.5995614
Faramarzi, E., Rajan, D., Fernandes, F.C.A., Christensen, M.P.: Blind super resolution of real-life video sequences. IEEE Trans. Image Process. 25(4), 1544–1555 (2016). https://doi.org/10.1109/TIP.2016.2523344
Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Gr. Appl. 22(2), 56–65 (2002). https://doi.org/10.1109/38.988747
Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25, 1423–1468 (2014). https://doi.org/10.1007/s00138-014-0623-4
Singh, A., Singh, J.: Survey on single image based super-resolution—implementation challenges and solutions. Multimed. Tools Appl. 79(3), 1641–1672 (2019). https://doi.org/10.1007/s11042-019-08254-0
Shahar, O., Faktor, A., Irani, M.: Space-time super-resolution from a single video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3353–3360, (2011). https://doi.org/10.1109/CVPR.2011.5995360
Son, S. et al.: NTIRE 2021 challenge on video super-resolution. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 166–181, (2021). https://doi.org/10.1109/CVPRW53098.2021.00026
Xiang, X., Tian, Y., Zhang, Y., Fu, Y., Allebach, J.P., Xu, C.: Zooming slow-mo: fast and accurate one-stage space-time video super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, pp. 3367–3376, (2020). https://doi.org/10.1109/CVPR42600.2020.00343
Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task-oriented flow. Int. J. Comput. Vis. 127(8), 1106–1125 (2017). https://doi.org/10.1007/s11263-018-01144-2
Kappeler, A., Yoo, S., Dai, Q., Katsaggelos, A.K.: Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2(2), 109–122 (2016). https://doi.org/10.1109/TCI.2016.2532323
Caballero, J. et al.: Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, vol. 2017-January, pp. 2848–2857, (2016), https://doi.org/10.1109/CVPR.2017.304
Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1664–1673, (2018). https://doi.org/10.1109/CVPR.2018.00179
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. (2018), Accessed: May 14, 2023. [Online]. Available: https://arxiv.org/abs/1802.08797v2
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Yun, F.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), pp. 294–310. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18
Tian, Y., Zhang, Y., Fu, Y., Xu, C.: TDAN: temporally deformable alignment network for video super-resolution. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3357–3366, (2018). https://doi.org/10.1109/CVPR42600.2020.00342
Wang, Z., Chen, J., Hoi, S.C.H.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern. Anal. Mach. Intell. (2020). https://doi.org/10.1109/tpami.2020.2982166
Jo, Y., Oh, W.S., Kang, J., Kim, S.J.: Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3224–3232, (2018). https://doi.org/10.1109/CVPR.2018.00340
Isobe, T., et al.: Video super-resolution with temporal group attention. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 8005–8014, (2020). https://doi.org/10.1109/CVPR42600.2020.00803
Kawulok, M., Benecki, P., Piechaczek, S., Hrynczenko, K., Kostrzewa, D., Nalepa, J.: Deep learning for multiple-image super-resolution. IEEE Geosci. Remote Sens. Lett. 17(6), 1062–1066 (2020). https://doi.org/10.1109/LGRS.2019.2940483
Salvetti, F., Mazzia, V., Khaliq, A., Chiaberge, M.: Multi-image super resolution of remotely sensed images using residual attention deep neural networks. Remote Sens. 12(14), 2207 (2020). https://doi.org/10.3390/rs12142207
Molini, A.B., Valsesia, D., Fracastoro, G., Magli, E.: DeepSUM: deep neural network for super-resolution of unregistered multitemporal images. IEEE Trans. Geosci. Remote Sens. 58(5), 3644–3656 (2019). https://doi.org/10.1109/TGRS.2019.2959248
Shechtman, E., Caspi, Y., Irani, M.: Space-time super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (2005). https://doi.org/10.1109/TPAMI.2005.85
Faramarzi, E., Rajan, D., Christensen, M.P.: Space-time super-resolution from multiple-videos. In: 2012 11th International Conference on Information Science, Signal Processing and their Applications, ISSPA 2012, (2012). https://doi.org/10.1109/ISSPA.2012.6310553
Mudenagudi, U., Banerjee, S., Kalra, P.K.: Space-time super-resolution using graph-cut optimization. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 995–1008 (2011). https://doi.org/10.1109/TPAMI.2010.167
Caspi, Y., Irani, M.: Spatio-temporal alignment of sequences. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1409–1424 (2002). https://doi.org/10.1109/TPAMI.2002.1046148
Takeda, H., Milanfar, P., Protter, M., Elad, M.: Super-resolution without explicit subpixel motion estimation. IEEE Trans. Image Process. 18(9), 1958–1975 (2009). https://doi.org/10.1109/TIP.2009.2023703
Faramarzi, E., Rajan, D., Christensen, M.P.: Space-time super-resolution from multiple-videos. In: The 11th International Conference on Information Sciences, Signal Processing and their Applications, pp. 23–28, (2012)
Gonzalez, R.C., Woods, R.E.: Digital image processing, 4th edn. Pearson, London (2018)
Burrus, C.S.: Iterative reweighted least squares. OpenStax CNX module: m45285, (2014)
Shewchuk, J.R.: An introduction to the conjugate gradient method without the agonizing Pain, Edition 1 ¼. In: School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, (1994)
Tao, X., Gao, H., Liao, R., Wang, J., Jia, J.: Detail-revealing Deep Video Super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 2017-October, pp. 4482–4490, (2017). https://doi.org/10.1109/ICCV.2017.479
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861
Takeda, H., Milanfar, P.: Removing motion blur with space-time processing. IEEE Trans. Image Process. 20(10), 2990–3000 (2011). https://doi.org/10.1109/TIP.2011.2131666
Cho, S., Lee, S.: Fast motion deblurring, p. 1. ACM Transactions on Graphics, ACM Press, New York (2009)
Deblurring–imageupscaler. https://imageupscaler.com/deblurring/. Accessed 01 Jan 2021
Funding
There is no funding or grant supporting this manuscript.
Author information
Authors and Affiliations
Contributions
Both authors contributed to all parts of the manuscripts.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interest associated with this study.
Ethical approval
This research is not sponsored by any industrial company and does not aim to endorse any commercial product.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file1 (MP4 925 KB)
Supplementary file2 (MP4 1444 KB)
Supplementary file3 (MP4 62 KB)
Supplementary file4 (MP4 681 KB)
Supplementary file5 (MP4 2712 KB)
Supplementary file6 (MP4 13960 KB)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Gharibi, Z., Faramarzi, S. Multi-frame spatio-temporal super-resolution. SIViP 17, 4415–4424 (2023). https://doi.org/10.1007/s11760-023-02675-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-023-02675-z