Skip to main content
SpringerLink
Log in

Deep Feature Fusion Network for Compressed Video Super-Resolution

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

The majority of conventional video super-resolution algorithms aim at reconstructing low-resolution videos after down-sampling. However, numerous low-resolution videos will be further compressed to adapt to the limited storage size and transmission bandwidth, leading to further video quality degradation. Significantly, the noise brought by compression often has a strong correlation with the content of the video frame itself. If we super-resolve compressed video frames directly, the noise may be amplified, leading to loss of important information or lower super-resolution performance. To ease those problems, we present an end-to-end deep feature fusion network with ordinary differential equation and dual attention mechanism for joint video compression artifacts reduction and super-resolution. The proposed network commendably enhances the spatial-temporal features fusion of different depths, improves the acquisition of meaningful information ability, and perfects reconstruction quality. In addition, we leverage several skip connections to fuse the captured in-depth feature information and the shallow to prevent information loss. The experimental results show that our proposed method is competent to reduce bit-rate and have excellent quality improvement effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Anoop V, Bipin PR (2020) Super-resolution based automatic diagnosis of retinal disease detection for clinical applications. Neural Process Lett 52(2):1155–1170

    Article  Google Scholar 

  2. Brandi F, de Queiroz R, Mukherjee D (2008) Super-resolution of video using key frames and motion estimation. In: 2008 15th IEEE international conference on image processing. IEEE, pp 321–324

  3. Cao H, Liu X, Wang Y, Li Y, Lei W (2020) Enhanced down/up-sampling-based video coding using the residual compensation. In: 2020 5th international conference on computer and communication systems (ICCCS). IEEE, pp 286–290

  4. Chang B, Meng L, Haber E, Ruthotto L, Begert D, Holtham E (2018) Reversible architectures for arbitrarily deep residual neural networks. In: Proceedings of the AAAI conference on artificial intelligence, vol 32

  5. Dai Y, Liu D, Wu F (2017) A convolutional neural network approach for post-processing in hevc intra coding. In: International conference on multimedia modeling. Springer, pp 28–39

  6. Feng L, Zhang X, Zhang X, Wang S, Wang R, Ma S (2018) A dual-network based super-resolution for compressed high definition video. In: Pacific Rim conference on multimedia. Springer, pp 600–610

  7. Fuoli D, Gu S, Timofte R (2019) Efficient video super-resolution through recurrent latent space propagation. In: 2019 IEEE/CVF international conference on computer vision workshop (ICCVW). IEEE, pp 3476–3485

  8. Georgis G, Lentaris G, Reisis D (2015) Reduced complexity superresolution for low-bitrate video compression. IEEE Trans Circuits Syst Video Technol 26(2):332–345

    Article  Google Scholar 

  9. Guan Z, Xing Q, Xu M, Yang R, Liu T, Wang, Z (2019) Mfqe 2.0: a new approach for multiframe quality enhancement on compressed video. IEEE Trans Pattern Anal Mach Intell 43(3): 949–963

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  11. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision. Springer, pp 630–645

  12. Ho MM, He G, Wang Z, Zhou J (2020) Down-sampling based video coding with degradation-aware restoration-reconstruction deep neural network. In: International conference on multimedia modeling. Springer, pp 99–110

  13. Ho MM, Zhou J, He G (2021) Rr-dncnn v2. 0: enhanced restoration–reconstruction deep neural network for down-sampling-based video coding. IEEE Trans Image Process 30:1702–1715

    Article  Google Scholar 

  14. Ho MM, Zhou J, He G, Li M, Li L (2020) Sr-cl-dmc: P-frame coding with super-resolution, color learning, and deep motion compensation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 124–125

  15. Hoang TM, Zhou J (2019) B-drrn: a block information constrained deep recursive residual network for video compression artifacts reduction. In: 2019 picture coding symposium (PCS). IEEE, pp 1–5

  16. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  17. Isobe T, Jia X, Gu S, Li S, Wang S, Tian Q (2020) Video super-resolution with recurrent structure-detail network. In: European conference on computer vision. Springer, pp 645–660

  18. Isobe T, Li S, Jia X, Yuan S, Slabaugh G, Xu C, Li YL, Wang S, Tian Q (2020) Video super-resolution with temporal group attention. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8008–8017

  19. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

  20. Köhler T, Huang X, Schebesch F, Aichert A, Maier A, Hornegger J (2016) Robust multiframe super-resolution employing iteratively re-weighted minimization. IEEE Trans Comput Imaging 2(1):42–58

    Article  MathSciNet  Google Scholar 

  21. Larsson G, Maire M, Shakhnarovich G (2016) Fractalnet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648

  22. Lee JK, Kim N, Cho S, Kang JW (2020) Deep video prediction network-based inter-frame coding in hevc. IEEE Access 8:95906–95917

    Article  Google Scholar 

  23. Li C, Zhang B, Hu H, Dai J (2019) Enhanced bird detection from low-resolution aerial image using deep neural networks. Neural Process Lett 49(3):1021–1039

    Article  Google Scholar 

  24. Li F, Bai H, Zhao Y (2020) Learning a deep dual attention network for video super-resolution. IEEE Trans Image Process 29:4474–4488

    Article  Google Scholar 

  25. Li X, Hu Y, Gao X, Tao D, Ning B (2010) A multi-frame image super-resolution method. Signal Process 90(2):405–414

    Article  Google Scholar 

  26. Li Y, Liu D, Li H, Li L, Li Z, Wu F (2018) Learning a convolutional neural network for image compact-resolution. IEEE Trans Image Process 28(3):1092–1107

    Article  MathSciNet  Google Scholar 

  27. Li Y, Liu D, Li H, Li L, Wu F, Zhang H, Yang H (2017) Convolutional neural network-based block up-sampling for intra frame coding. IEEE Trans Circuits Syst Video Technol 28(9):2316–2330

    Article  Google Scholar 

  28. Liao R, Tao X, Li R, Ma Z, Jia J (2015) Video super-resolution via deep draft-ensemble learning. In: Proceedings of the IEEE international conference on computer vision, pp 531–539

  29. Lin J, Liu D, Yang H, Li H, Wu F (2018) Convolutional neural network-based block up-sampling for hevc. IEEE Trans Circuits Syst Video Technol 29(12):3701–3715

    Article  Google Scholar 

  30. Liu C, Sun D (2013) On Bayesian adaptive video super resolution. IEEE Trans Pattern Anal Mach Intell 36(2):346–360

    Article  Google Scholar 

  31. Lu Y, Zhong A, Li Q, Dong B (2018) Beyond finite layer neural networks: bridging deep architectures and numerical differential equations. In: International conference on machine learning. PMLR, pp 3276–3285

  32. Ma Z, Liao R, Tao X, Xu L, Jia J, Wu E (2015) Handling motion blur in multi-frame super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5224–5232

  33. Mnih V, Heess N, Graves A et al (2014) Recurrent models of visual attention. In: Advances in neural information processing systems, pp 2204–2212

  34. Molina R, Katsaggelos AK, Alvarez L, Mateos J (2006) Toward a new video compression scheme using super-resolution. In: Visual communications and image processing 2006, vol 6077. International Society for Optics and Photonics, p 607706

  35. Peng H, Chen X, Zhao J (2020) Residual pixel attention network for spectral reconstruction from RGB images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 486–487

  36. Pourazad MT, Doutre C, Azimi M, Nasiopoulos P (2012) Hevc: the new gold standard for video compression: how does hevc compare with h.264/avc? IEEE Consum Electron Mag 1(3):36–46

    Article  Google Scholar 

  37. Shen M, Xue P, Wang C (2011) Down-sampling based video coding using super-resolution technique. IEEE Trans Circuits Syst Video Technol 21(6):755–765

    Article  Google Scholar 

  38. Soh JW, Park J, Kim Y, Ahn B, Lee HS, Moon YS, Cho NI (2018) Reduction of video compression artifacts based on deep temporal networks. IEEE Access 6:63094–63106

    Article  Google Scholar 

  39. Tang T, Li L (2016) Adaptive deblocking method for low bitrate coded hevc video. J Vis Commun Image Represent 38:721–734

    Article  Google Scholar 

  40. Tian Y, Zhang Y, Fu Y, Xu C (2020) Tdan: temporally-deformable alignment network for video super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3360–3369

  41. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  42. Wang L, Guo Y, Lin Z, Deng X, An W (2018) Learning for video super-resolution through HR optical flow estimation. In: Asian conference on computer vision. Springer, pp 514–529

  43. Wang L, Guo Y, Liu L, Lin Z, Deng X, An W (2020) Deep video super-resolution using HR optical flow estimation. IEEE Trans Image Process 29:4323–4336

    Article  Google Scholar 

  44. Weinan E (2017) A proposal on machine learning via dynamical systems. Commun Math Stat 5(1):1–11

    Article  MathSciNet  Google Scholar 

  45. Yang R, Xu M, Wang Z, Li T (2018) Multi-frame quality enhancement for compressed video. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6664–6673

  46. Ying X, Wang L, Wang Y, Sheng W, An W, Guo Y (2020) Deformable 3d convolution for video super-resolution. IEEE Signal Process Lett 27:1500–1504

    Article  Google Scholar 

  47. Zhang X, Dong H, Hu Z, Lai WS, Wang F, Yang MH (2018) Gated fusion network for joint image deblurring and super-resolution. arXiv preprint arXiv:1807.10806

  48. Zhang X, Li Z, Change Loy C, Lin D (2017) Polynet: a pursuit of structural diversity in very deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 718–726

  49. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp 286–301

  50. Zhao H, Kong X, He J, Qiao Y, Dong C (2020) Efficient image super-resolution using pixel attention. In: European conference on computer vision. Springer, pp 56–72

  51. Zhou Y, Sun X, Zha ZJ, Zeng W (2018) Mict: mixed 3d/2d convolutional tube for human action recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 449–458

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 61871279 & 62081330105 and in part by the Fundamental Research Funds for the Central Universities under Grant 2021SCU12061.

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Sichuan University, Chengdu, 610065, Sichuan, China

    Yue Wang, Xiaohong Wu, Xiaohai He, Chao Ren & Tingrong Zhang

Authors
  1. Yue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaohong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaohai He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chao Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tingrong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaohai He.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Wu, X., He, X. et al. Deep Feature Fusion Network for Compressed Video Super-Resolution. Neural Process Lett 54, 4427–4441 (2022). https://doi.org/10.1007/s11063-022-10816-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-10816-7

Keywords

Search

Navigation