Skip to main content
SpringerLink
Log in

Real-EVE: Real-Time Edge-Assist Video Enhancement for Joint Denoising and Super-Resolution

  • Conference paper
  • First Online:
Algorithms and Architectures for Parallel Processing (ICA3PP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14487))

  • 104 Accesses

Abstract

Real-time video applications have received much attention in recent years. However, the perceived quality of real-time videos in many situations is far from ideal due to two major obstacles: noise in the video frames caused by limited camera hardware and low resolution caused by bandwidth-limited networks. A straightforward solution is a direct investment in photography and networking hardware, but it is obviously cost-ineffective and unscalable. We are motivated to develop an alternative solution by leveraging edge AI. We propose a new Real-time Edge-assist Video Enhancement (Real-EVE) framework. It includes two key designs: The video-enhancement deep neural network (VE-DNN), which jointly eliminates noise and super-resolves videos in real time with a small inference delay; and the video-enhancement-aware adaptive bitrate streaming (VEA-ABR), which adapts sending rate in response to changing network conditions to optimize the video quality posterior to video enhancement. We develop a real-world prototype of the proposed Real-EVE, demonstrating Real-EVE outperforms all benchmarks, and both the VE-DNN and VEA-ABR bring drastic performance gain.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Akhshabi, S., Begen, A.C., Dovrolis, C.: An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP. In: MMSys (2011)

    Google Scholar 

  2. Chan, K.C., Zhou, S., Xu, X., Loy, C.C.: BasicVSR++: improving video super-resolution with enhanced propagation and alignment. In: CVPR (2022)

    Google Scholar 

  3. Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR, pp. 3291–3300 (2018)

    Google Scholar 

  4. Chen, H., Jin, Y., Xu, K., Chen, Y., Zhu, C.: Multiframe-to-multiframe network for video denoising. IEEE (2021)

    Google Scholar 

  5. Dash.js: Dash.js. https://github.com/Dash-Industry-Forum/dash.js/wiki (2023)

  6. Emmons, H., Vairaktarakis, G.: Flow shop scheduling: theoretical results, algorithms, and applications, vol. 182. Springer Science & Business Media (2012)

    Google Scholar 

  7. Engiz, B.K., Kurnaz, Ç.: Comparison of signal strengths of 2G/3G/4G services on a university campus. Int. J. Appl. Math. Electron. Comput. (Special Issue-1), 37–42 (2016)

    Google Scholar 

  8. Gow, R.D., et al.: A comprehensive tool for modeling CMOS image-sensor-noise performance. vol. 54, pp. 1321–1329. IEEE (2007)

    Google Scholar 

  9. Houdaille, R., Gouache, S.: Shaping HTTP adaptive streams for a better user experience. In: MMSys, pp. 1–9 (2012)

    Google Scholar 

  10. Hung, C.C., Ananthanarayanan, G., Bodik, P., Golubchik, L., Yu, M., Bahl, P., Philipose, M.: Videoedge: Processing camera streams using hierarchical clusters. In: SEC, pp. 115–131. IEEE (2018)

    Google Scholar 

  11. Huynh-Thu, Q., Ghanbari, M.: Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 44(13), 800–801 (2008)

    Article  Google Scholar 

  12. Isobe, T., Zhu, F., Jia, X., Wang, S.: Revisiting temporal modeling for video super-resolution. In: BMVC (2020)

    Google Scholar 

  13. ITU-T Recommendations: One-way transmission time. https://www.itu.int/rec/T-REC-G.114-200305-I/en (2023)

  14. Jacobson, V., Frederick, R., Casner, S., Schulzrinne, H.: Realtime transport protocol (RTP). https://www.ietf.org/rfc/rfc3550.txt (2014)

  15. Jansen, B., Goodwin, T., Gupta, V., Kuipers, F., Zussman, G.: Performance evaluation of WebRTC-based video conferencing. SIGMETRICS 45(3), 56–68 (2018)

    Article  Google Scholar 

  16. Jiang, J., Sekar, V., Zhang, H.: Improving fairness, efficiency, and stability in HTTP-based adaptive video streaming with FESTIVE. In: CoNEXT (2012)

    Google Scholar 

  17. Kong, X., Kong, X., et al.: Real-time mask identification for COVID-19: an edge-computing-based deep learning framework. IEEE Internet Things J. 8(21), 15929–15938 (2021)

    Article  Google Scholar 

  18. Lee, R., Venieris, S.I., Lane, N.D.: Deep neural network-based enhancement for image and video streaming systems: a survey and future directions. ACM Comput. Surv. 54(8), 1–30 (2021)

    Google Scholar 

  19. Liu, C., Yang, H., Fu, J., Qian, X.: Learning trajectory-aware transformer for video super-resolution. In: CVPR, pp. 5687–5696 (2022)

    Google Scholar 

  20. Mao, H., Netravali, R., Alizadeh, M.: Neural adaptive video streaming with pensieve. In: SIGCOMM (2017)

    Google Scholar 

  21. Mok, R.K., Chan, E.W., Luo, X., Chang, R.K.: Inferring the QoE of HTTP video streaming from user-viewing activities. In: SIGCOMM W-MUST, pp. 31–36 (2011)

    Google Scholar 

  22. Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: CVPR (2016)

    Google Scholar 

  23. Setiadi, D.R.I.M.: PSNR vs SSIM: imperceptibility quality assessment for image steganography. Multimed. Tools Appl. 80(6), 8423–8444 (2021)

    Article  Google Scholar 

  24. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR, pp. 1874–1883 (2016)

    Google Scholar 

  25. Sun, L., Zong, T., Liu, Y., Wang, Y., Zhu, H.: Optimal strategies for live video streaming in the low-latency regime. In: ICNP, pp. 1–4. IEEE (2019)

    Google Scholar 

  26. Tassano, M., Delon, J., Veit, T.: DVDnet: A fast network for deep video denoising. In: ICIP (2019)

    Google Scholar 

  27. Tassano, M., Delon, J., Veit, T.: FastDVDnet: towards real-time deep video denoising without flow estimation. In: CVPR (2020)

    Google Scholar 

  28. Viola, R., Martin, A., Zorrilla, M., Montalbán, J.: MEC proxy for efficient cache and reliable multi-CDN video distribution. In: IEEE BMSB (2018)

    Google Scholar 

  29. Wei, K., Fu, Y., Yang, J., Huang, H.: A physics-based noise formation model for extreme low-light raw denoising. In: CVPR (2020)

    Google Scholar 

  30. Xiph.org: Derf’s test media collection. https://media.xiph.org/video/derf (2022)

  31. Yeo, H., Jung, Y., Kim, J., Shin, J., Han, D.: Neural adaptive content-aware internet video delivery. In: USENIX OSDI, pp. 645–661 (2018)

    Google Scholar 

  32. Yin, X., Jindal, A., Sekar, V., Sinopoli, B.: A control-theoretic approach for dynamic adaptive video streaming over HTTP. In: SIGCOMM (2015)

    Google Scholar 

  33. Zhang, H., Ananthanarayanan, G., Bodik, P., Philipose, M., Bahl, P., Freedman, M.J.: Live video analytics at scale with approximation and delay-tolerance. In: USENIX NSDI (2017)

    Google Scholar 

  34. Zuo, X., Yang, J., Wang, M., Cui, Y.: Adaptive bitrate with user-level QoE preference for video streaming. In: INFOCOM, pp. 1279–1288. IEEE (2022)

    Google Scholar 

Download references

Acknowledgements

This research was supported in part by the Toronto Mobility Scheme of the University of Sydney.

Author information

Authors and Affiliations

  1. Faculty of Engineering, The University of Sydney, Sydney, Australia

    Liming Ge, Wei Bao, Dong Yuan & Bing Bing Zhou

Authors
  1. Liming Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dong Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bing Bing Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liming Ge .

Editor information

Editors and Affiliations

  1. Royal Melbourne Institute of Technology, Melbourne, VIC, Australia

    Zahir Tari

  2. Tianjin University, Tianjin, China

    Keqiu Li

  3. University of Arizona, Tucson, AZ, USA

    Hongyi Wu

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ge, L., Bao, W., Yuan, D., Zhou, B.B. (2024). Real-EVE: Real-Time Edge-Assist Video Enhancement for Joint Denoising and Super-Resolution. In: Tari, Z., Li, K., Wu, H. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2023. Lecture Notes in Computer Science, vol 14487. Springer, Singapore. https://doi.org/10.1007/978-981-97-0834-5_19

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-0834-5_19

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-0833-8

  • Online ISBN: 978-981-97-0834-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation