Skip to main content
SpringerLink
Log in
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 Workshops pp 602–618Cite as

EESRNet: A Network for Energy Efficient Super-Resolution

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13802))

Abstract

Recently, various high-performance video super-resolution methods have been proposed. However, deployment on mobile phones is cumbersome due to the limitations of mobile phones’ power consumption and computing power. We find methods that exploit temporal information in videos (e.g. optical flow) require huge energy consumption. Therefore, we use hidden features to preserve temporal information. Besides, the energy-efficient super-resolution network (EESRNet) is obtained by removing the residual connections in the Anchor-Based Plain Network (ABPN) [8]. Combining the two, we propose a Temporal Energy Efficient Super-Resolution Network (TEESRNet), which can efficiently utilize video spatio-temporal information with low energy consumption. Experiments show that for EESRNet, compared with ABPN, the latency is reduced by more than \(40\%\), while performance decreases slightly. Furthermore, for TEESRNet, the PSNR is improved by 0.24 dB and 1.19 dB compared to EESRNet and RRN [19] respectively, while still maintaining real-time (<30 ms).

S. Yue, C. Li and Z. Zhuge—These authors contributed equally to this work.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://github.com/MediaTek-NeuroPilot/mai22-real-time-video-sr.

  2. 2.

    https://github.com/lchia/EESRNet.

References

  1. Benmeziane, H., Maghraoui, K.E., Ouarnoughi, H., Niar, S., Wistuba, M., Wang, N.: A comprehensive survey on hardware-aware neural architecture search. arXiv preprint arXiv:2101.09336 (2021)

  2. Bouzidi, H., Ouarnoughi, H., Niar, S., Cadi, A.A.E.: Performance prediction for convolutional neural networks on edge GPUs. In: CF 2021, Association for Computing Machinery, pp. 54–62. New York (2021). https://doi.org/10.1145/3457388.3458666

  3. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Chapter  Google Scholar 

  4. Caballero, J., et al.: Real-time video super-resolution with spatio-temporal networks and motion compensation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4778–4787 (2017)

    Google Scholar 

  5. Cheng, H., Liao, W., Tang, X., Yang, M.Y., Sester, M., Rosenhahn, B.: Exploring dynamic context for multi-path trajectory prediction. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 12795–12801 (2021). https://doi.org/10.1109/ICRA48506.2021.9562034

  6. Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4641–4650 (2021)

    Google Scholar 

  7. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13

    Chapter  Google Scholar 

  8. Du, Z., Liu, J., Tang, J., Wu, G.: Anchor-based plain net for mobile image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2494–2502 (2021)

    Google Scholar 

  9. Fuoli, D., Gu, S., Timofte, R.: Efficient video super-resolution through recurrent latent space propagation. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3476–3485. IEEE (2019)

    Google Scholar 

  10. Goodfellow, I., et al.: Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27 (2014)

    Google Scholar 

  11. Hong, C., Kim, H., Baik, S., Oh, J., Lee, K.M.: DAQ: channel-wise distribution-aware quantization for deep image super-resolution networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2675–2684 (2022)

    Google Scholar 

  12. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  13. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks. Adv. Neural Inf. Process. Syst. 29 (2016)

    Google Scholar 

  14. Ignatov, A., Romero, A., Kim, H., Timofte, R.: Real-time video super-resolution on smartphones with deep learning, mobile AI 2021 challenge: report. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2535–2544 (2021)

    Google Scholar 

  15. Ignatov, A., Timofte, R., Denna, M., Younes, A.: Real-time quantized image super-resolution on mobile NPUs, mobile AI 2021 challenge: report. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2525–2534 (2021)

    Google Scholar 

  16. Ignatov, A., et al.: AI benchmark: all about deep learning on smartphones in 2019. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3617–3635. IEEE (2019)

    Google Scholar 

  17. Ignatov, A., et al.: Real-time video super-resolution on mobile NPUs with deep learning, mobile AI & AIM 2022 challenge: report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)

    Google Scholar 

  18. Ignatov, A., Van Gool, L., Timofte, R.: Replacing mobile camera ISP with a single deep learning model. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2275–2285 (2020). https://doi.org/10.1109/CVPRW50498.2020.00276

  19. Isobe, T., Zhu, F., Jia, X., Wang, S.: Revisiting temporal modeling for video super-resolution. arXiv preprint arXiv:2008.05765 (2020)

  20. Jo, Y., Oh, S.W., Kang, J., Kim, S.J.: Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3224–3232 (2018)

    Google Scholar 

  21. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

    Google Scholar 

  22. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

    Google Scholar 

  23. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

    Google Scholar 

  24. Li, D., Liu, Y., Wang, Z.: Video super-resolution using non-simultaneous fully recurrent convolutional network. IEEE Trans. Image Process. 28(3), 1342–1355 (2018)

    Article  MathSciNet  Google Scholar 

  25. Li, S., He, F., Du, B., Zhang, L., Xu, Y., Tao, D.: Fast spatio-temporal residual network for video super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10522–10531 (2019)

    Google Scholar 

  26. Li, Y., et al.: NTIRE 2022 challenge on efficient super-resolution: methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1062–1102 (2022)

    Google Scholar 

  27. Liao, R., Tao, X., Li, R., Ma, Z., Jia, J.: Video super-resolution via deep draft-ensemble learning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 531–539 (2015)

    Google Scholar 

  28. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

    Google Scholar 

  29. Liu, J., Wang, Q., Zhang, D., Shen, L.: Super-resolution model quantized in multi-precision. Electronics 10(17), 2176 (2021)

    Article  Google Scholar 

  30. Marchisio, A., Massa, A., Mrazek, V., Bussolino, B., Martina, M., Shafique, M.: NASCaps: a framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks. In: 2020 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pp. 1–9. IEEE (2020)

    Google Scholar 

  31. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  32. Sun, B., Li, J., Shao, M., Fu, Y.: LRPRNet: lightweight deep network by low-rank pointwise residual convolution. In: IEEE Transactions on Neural Networks and Learning Systems, pp. 1–11 (2021). https://doi.org/10.1109/TNNLS.2021.3117685

  33. Wang, H., Chen, P., Zhuang, B., Shen, C.: Fully quantized image super-resolution networks. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 639–647 (2021)

    Google Scholar 

  34. Wang, P., Chen, W., He, X., Chen, Q., Liu, Q., Cheng, J.: Optimization-based post-training quantization with bit-split and stitching. IEEE Trans. Pattern Anal. Mach. Intell. 45(2), 2119–2135 (2022)

    Article  Google Scholar 

  35. Wang, P., He, X., Chen, Q., Cheng, A., Liu, Q., Cheng, J.: Unsupervised network quantization via fixed-point factorization. IEEE Trans. Neural Netw. Learn. Syst. 32(6), 2706–2720 (2020)

    Article  MathSciNet  Google Scholar 

  36. Wang, P., Hu, Q., Zhang, Y., Zhang, C., Liu, Y., Cheng, J.: Two-step quantization for low-bit neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4376–4384 (2018)

    Google Scholar 

  37. Wang, X., Chan, K.C., Yu, K., Dong, C., Change Loy, C.: EDVR: video restoration with enhanced deformable convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2019)

    Google Scholar 

  38. Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)

    Google Scholar 

  39. Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2011)

    Google Scholar 

  40. Xu, W., He, X., Zhao, T., Hu, Q., Wang, P., Cheng, J.: Soft threshold ternary networks. arXiv preprint arXiv:2204.01234 (2022)

  41. Yan, B., Lin, C., Tan, W.: Frame and feature-context video super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5597–5604 (2019)

    Google Scholar 

  42. Yang, R., et al.: NTIRE 2022 challenge on super-resolution and quality enhancement of compressed video: dataset, methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1221–1238 (2022)

    Google Scholar 

  43. Zhang, K., et al.: AIM 2020 challenge on efficient super-resolution: methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 5–40. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_1

    Chapter  Google Scholar 

  44. Zhang, X., Zeng, H., Zhang, L.: Edge-oriented convolution block for real-time super resolution on mobile devices. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4034–4043 (2021)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China (Grant No. 2021ZD0201504), and the National Natural Science Foundation of China (No. 62106267).

Author information

Authors and Affiliations

  1. North China University of Technology (NCUT), Beijing, China

    Shijie Yue & Ruixia Song

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

    Zhengyang Zhuge

  3. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Chenghua Li & Zhengyang Zhuge

  4. Nanjing Artificial Intelligence Research of IA (AiRiA), Nanjing, China

    Chenghua Li

Authors
  1. Shijie Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenghua Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhengyang Zhuge
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruixia Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Chenghua Li or Ruixia Song .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 190 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yue, S., Li, C., Zhuge, Z., Song, R. (2023). EESRNet: A Network for Energy Efficient Super-Resolution. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13802. Springer, Cham. https://doi.org/10.1007/978-3-031-25063-7_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25063-7_38

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25062-0

  • Online ISBN: 978-3-031-25063-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation