Skip to main content
SpringerLink
Log in

IAA-VSR: An iterative alignment algorithm for video super-resolution

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Video Super-Resolution (VSR) aims at producing a high-resolution video from its corresponding low-resolution input frames. In VSR, the key to generating high-quality output is to exploit the spatial similarity of temporal frames. Most VSR methods achieve this by super-resolving a single reference frame with the aid of multiple frames in a temporal window. For this goal, some alignment methods have been proposed to compensate for the motion between adjacent frames. However, these methods lack more upper-level and unified guidance to progressively align neighboring frames, which often leads to poor results when encountering large motions. In this paper, we propose a unified Iterative Alignment Algorithm (IAA) for more accurate frame alignment in VSR. In IAA, each adjacent frame only needs to be aligned to its nearest neighbor, which greatly eases the alignment problem for all kinds of motions. To show the effectiveness of our method, we apply IAA to red the Enhanced Deformable Video super-Resolution (EDVR) network and obtain a new network called IAA-VSR. Extensive experiments show that our IAA-VSR consistently improves the performance of EDVR on benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Kim J, Lee JK, Lee KM (2016) Deeply-recursive convolutional network for image super-resolution. In: CVPR. IEEE Computer Society, pp 1637–1645

  2. Lai W-S, Huang J-B, Ahuja N, Yang M-H (2017) Deep laplacian pyramid networks for fast and accurate super-resolution. In: CVPR. IEEE Computer Society, pp 5835–5843

  3. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In: CVPR. IEEE Computer Society, pp 2790–2798

  4. Haris M, Shakhnarovich G, Ukita N (2018) Deep back-projection networks for super-resolution. In: CVPR. IEEE Computer Society, pp 1664–1673

  5. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: ECCV (7). Lecture Notes in Computer Science, vol 11211. Springer, pp 294–310

  6. Caballero J, Ledig C, Aitken AP, Acosta A, Totz J, Wang Z, Shi W (2017) Real-time video super-resolution with spatio-temporal networks and motion compensation. In: CVPR. IEEE Computer Society, pp 2848–2857

  7. Kappeler A, Yoo S, Dai Q, Katsaggelos AK (2016) Video super-resolution with convolutional neural networks. IEEE Trans Comput Imaging 2(2):109–122

    Article  MathSciNet  Google Scholar 

  8. Liu D, Wang Z, Fan Y, Liu X, Wang Z, Chang S, Huang TS (2017) Robust video super-resolution with learned temporal dynamics. In: ICCV. IEEE Computer Society, pp 2526– 2534

  9. Xue T, Chen B, Wu J, Wei D, Freeman WT (2019) Video enhancement with task-oriented flow. Int J Comput Vis 127(8):1106–1125

    Article  Google Scholar 

  10. Jo Y, Oh SW, Kang J, Kim SJ (2018) Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: CVPR. IEEE Computer Society, pp 3224– 3232

  11. Wang X, Chan KCK, Yu K, Dong C, Loy CC (2019) EDVR: video restoration with enhanced deformable convolutional networks. In: CVPR workshops. Computer Vision Foundation / IEEE, pp 1954–1963

  12. Zhu X, Hu H, Lin S, Dai J (2019) Deformable convnets V2: more deformable, better results. In: CVPR. Computer Vision Foundation / IEEE, pp 9308–9316

  13. Tian Y, Zhang Y, Fu Y, Xu C (2020) TDAN: temporally-deformable alignment network for video super-resolution. In: CVPR. IEEE, pp 3357–3366

  14. Nah S, Timofte R, Baik S, Hong S, Moon G, Son S, Lee KM, et al. (2019) NTIRE 2019 challenge on video deblurring: Methods and results. In: CVPR workshops. Computer Vision Foundation / IEEE, pp 1974–1984

  15. Nah S, Timofte R, Gu S, Baik S, Hong S, Moon G, Son S, Lee KM et al (2019) NTIRE 2019 challenge on video super-resolution: Methods and results. In: CVPR workshops. Computer Vision Foundation / IEEE, pp 1985–1995

  16. Nah S, Baik S, Hong S, Moon G, Son S, Timofte R, Lee KM (2019) NTIRE 2019 challenge on video deblurring and super-resolution: Dataset and study. In: CVPR workshops. Computer Vision Foundation / IEEE, pp 1996–2005

  17. Freeman WT, Jones TR, Pasztor EC (2002) Example-based super-resolution. IEEE Comput Graph Appl 22(2):56–65

    Article  Google Scholar 

  18. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. In: ICCV. IEEE Computer Society, pp 349–356

  19. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process. 19(11):2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  20. Freedman G, Fattal R (2011) Image and video upscaling from local self-examples. ACM Trans Graph 30(2):12:1–12:11

    Article  Google Scholar 

  21. Timofte R, Smet VD, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: ICCV. IEEE Computer Society, pp 1920–1927

  22. Timofte R, Smet VD, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: ACCV (4). Lecture Notes in Computer Science, vol 9006. Springer, pp 111–126

  23. Schulter S, Leistner C, Bischof H (2015) Fast and accurate image upscaling with super-resolution forests. In: CVPR. IEEE Computer Society, pp 3791–3799

  24. Yang J, Wang Z, Lin Z, Cohen S, Huang TS (2012) Coupled dictionary training for image super-resolution. IEEE Trans Image Process 21(8):3467–3478

    Article  MathSciNet  MATH  Google Scholar 

  25. Pérez-Pellitero E, Salvador J, Hidalgo JR, Rosenhahn B (2016) Psyco: Manifold span reduction for super resolution. In: CVPR. IEEE Computer Society, pp 1837–1845

  26. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. In: CVPR. IEEE Computer Society, pp 1646–1654

  27. Ledig C, Theis L, Huszar F, Caballero J, Cunningham A, Acosta A, Aitken AP, Tejani A, Totz J, Wang Z, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR. IEEE Computer Society, pp 105–114

  28. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image super-resolution. In: CVPR workshops. IEEE Computer Society, pp 1132–1140

  29. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: CVPR. IEEE Computer Society, pp 2472–2481

  30. Liu J, Zhang W, Tang Y, Tang J, Wu G (2020) Residual feature aggregation network for image super-resolution. In: CVPR. IEEE, pp 2356–2365

  31. Dai T, Cai J, Zhang Y, Xia S-T, Zhang L (2019) Second-order attention network for single image super-resolution. In: CVPR. Computer Vision Foundation / IEEE, pp 11065–11074

  32. Guo Y, Chen J, Wang J, Chen Q, Cao J, Deng Z, Xu Y, Tan M (2020) Closed-loop matters: Dual regression networks for single image super-resolution. In: CVPR. IEEE, pp 5406–5415

  33. Dong C, Loy CC, He K, Tang X (2014) Learning a deep convolutional network for image super-resolution. In: ECCV (4). Lecture Notes in Computer Science, vol 8692. Springer, pp 184–199

  34. Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR. IEEE Computer Society, pp 1874–1883

  35. Timofte R, Agustsson E, Gool LV, Yang M-H, et al. (2017) NTIRE 2017 challenge on single image super-resolution: Methods and results. In: CVPR workshops. IEEE Computer Society, pp 1110–1121

  36. Kim SY, Lim J, Na T, Kim M (2018) 3dsrnet: Video super-resolution using 3d convolutional neural networks. CoRR abs/1812.09079

  37. Yi P, Wang Z, Jiang K, Jiang J, Ma J (2019) Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. In: ICCV. IEEE, pp 3106–3115

  38. Huang Y, Wang W, Wang L (2015) Bidirectional recurrent convolutional networks for multi-frame super-resolution. In: NIPS, pp 235–243

  39. Isobe T, Jia X, Gu S, Li S, Wang S, Tian Q (2020) Video super-resolution with recurrent structure-detail network. In: ECCV (12). Lecture Notes in Computer Science, vol 12357. Springer, pp 645–660

  40. Li W, Tao X, Guo T, Qi L, Lu J, Jia J (2020) Mucan: Multi-correspondence aggregation network for video super-resolution. In: ECCV (10). Lecture Notes in Computer Science, vol 12355. Springer, pp 335–351

  41. Caballero J, Ledig C, Aitken AP, Acosta A, Totz J, Wang Z, Shi W (2017) Real-time video super-resolution with spatio-temporal networks and motion compensation. In: CVPR. IEEE Computer Society, pp 2848–2857

  42. Tao X, Gao H, Liao R, Wang J, Jia J (2017) Detail-revealing deep video super-resolution. In: ICCV. IEEE Computer Society, pp 4482–4490

  43. Haris M, Shakhnarovich G, Ukita N (2019) Recurrent back-projection network for video super-resolution. In: CVPR. Computer Vision Foundation / IEEE, pp 3897–3906

  44. Chan KCK, Wang X, Yu K, Dong C, Loy CC (2020) Understanding deformable alignment in video super-resolution. CoRR abs/2009.07265

  45. Isobe T, Li S, Jia X, Yuan S, Slabaugh GG, Xu C, Li Y-L, Wang S, Tian Q (2020) Video super-resolution with temporal group attention. In: CVPR. IEEE, pp 8005–8014

  46. Liu C, Sun D (2014) On bayesian adaptive video super resolution. IEEE Trans Pattern Anal Mach Intell 36(2):346–360

    Article  Google Scholar 

  47. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  48. Yi P, Wang Z, Jiang K, Jiang J, Ma J (2019) Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. In: ICCV. IEEE, pp 3106–3115

  49. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: ICLR (poster)

  50. Liao R, Tao X, Li R, Ma Z, Jia J (2015) Video super-resolution via deep draft-ensemble learning. In: ICCV. IEEE Computer Society, pp 531–539

  51. Sajjadi MSM, Vemulapalli R, Brown M (2018) Frame-recurrent video super-resolution. In: CVPR. IEEE Computer Society, pp 6626–6634

Download references

Acknowledgements

This paper is supported by the program B for Outstanding PhD candidate of Nanjing University.

Author information

Authors and Affiliations

  1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China

    Jie Liu, Jie Tang & Gangshan Wu

Authors
  1. Jie Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gangshan Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Tang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, J., Tang, J. & Wu, G. IAA-VSR: An iterative alignment algorithm for video super-resolution. Appl Intell 52, 16572–16585 (2022). https://doi.org/10.1007/s10489-022-03364-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03364-z

Keywords

Search

Navigation