research-article

Revisiting super-resolution for internet video streaming

Authors:

Yi QinAuthors Info & Claims

NOSSDAV '22: Proceedings of the 32nd Workshop on Network and Operating Systems Support for Digital Audio and Video

Pages 8 - 14

https://doi.org/10.1145/3534088.3534344

Published: 11 July 2022 Publication History

Abstract

Recent advancements of neural-enhanced techniques, especially super-resolution (SR), show great potential in revolutionizing the landscape of Internet video delivery. However, there are still quite a few key questions (e.g., how to choose a proper resolution configuration for training samples, how to set the training patch size, how to perform the best patch selection, how to set the update frequency of SR model) that have not been well investigated and understood. In this paper, we perform a dedicated measurement study to revisit super-resolution techniques for Internet video streaming. Our measurements are based on real-world video datasets, and the results provide a number of important insights: (1) It is possible that the SR model trained with low-resolution patches (e.g., (540p, 1080p) pairs) can achieve almost the same performance as that trained with high-resolution patches (e.g., (1080p, 2160p) pairs); (2) Compared to the saliency of training patches, the size of training patches has little impact on the performance of trained SR model; (3) The improvement of video quality brought by more frequent SR model update is not very significant. We also discuss the implications of our findings for system design, and we believe that our work is essential for paving the way for the success of future neural-enhanced video streaming systems.

References

[1]

Eirikur Agustsson and Radu Timofte. 2017. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 126--135.

[2]

Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. 2018. Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network. arXiv preprint arXiv:1803.08664 (2018).

[3]

Duin Baek, Mallesham Dasari, Samir R Das, and Jihoon Ryoo. 2021. dcSR: practical video quality enhancement using data-centric super resolution. In Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies. 336--343.

Digital Library

[4]

Raghav Bansal, Gaurav Raj, and Tanupriya Choudhury. 2016. Blur image detection using Laplacian operator and Open-CV. In 2016 International Conference System Modeling & Advancement in Research Trends (SMART). IEEE, 63--67.

[5]

Nabajeet Barman and Maria G Martini. 2019. QoE modeling for HTTP adaptive video streaming-a survey and open challenges. Ieee Access 7 (2019), 30831--30859.

[6]

Cisco. Accessed March 3, 2022. Cisco Annual Internet Report (2018--2023) White Paper. https://www.cisco.com/c7en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html.

[7]

Mallesham Dasari, Arani Bhattacharya, Santiago Vargas, Pranjal Sahu, Aruna Balasubramanian, and Samir R. Das. 2020. Streaming 360-Degree Videos Using Super-Resolution. In Proc. of IEEE Conference on Computer Communications (INFOCOM). 1977--1986.

[8]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In European conference on computer vision. Springer, 184--199.

[9]

Chao Dong, Chen Change Loy, and Xiaoou Tang. 2016. Accelerating the super-resolution convolutional neural network. In European conference on computer vision. Springer, 391--407.

[10]

Facebook. Accessed March 2, 2021. Facebook Live. https://www.facebook.com/formedia/solutions/facebook-live.

[11]

Khizar Hayat. 2018. Multimedia super-resolution via deep learning: A survey. Digital Signal Processing 81 (2018), 198--217.

Digital Library

[12]

Robert Keys. 1981. Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing 29, 6 (1981), 1153--1160.

[13]

Jaehong Kim, Youngmok Jung, Hyunho Yeo, Juncheol Ye, and Dongsu Han. 2020. Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning. In Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication on the Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM). 107--125.

[14]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1646--1654.

[15]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1637--1645.

[16]

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4681--4690.

[17]

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 136--144.

[18]

Limelight. Accessed March 3, 2022. MARKET RESEARCH: THE STATE OF ONLINE VIDEO 2019. https://www.limelight.com/resources/white-paper/state-of-online-video-2019/.

[19]

Zhicong Lu, Haijun Xia, Seongkook Heo, and Daniel Wigdor. 2018. You watch, you give, and you engage: a study of live streaming practices in China. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1--13.

Digital Library

[20]

Zhenxiao Luo, Zelong Wang, Jinyu Chen, Miao Hu, Yipeng Zhou, Tom Z. J. Fu, and Di Wu. 2021. CrowdSR: Enabling High-Quality Video Ingest in Crowd-sourced Livecast via Super-Resolution. In Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV). 90--97.

Digital Library

[21]

opencv. Accessed June 1, 2021. opencv-python. https://github.com/opencv/opencv-python.

[22]

Pytorch. Accessed June 1, 2021. Pytorch. https://pytorch.org/.

[23]

Jing Tian and Kai-Kuang Ma. 2011. A survey on super-resolution imaging. Signal, Image and Video Processing 5, 3 (2011), 329--342.

[24]

Twitch TV. Accessed March 2, 2021. Twitch TV. https://www.twitch.tv/.

[25]

Zhihao Wang, Jian Chen, and Steven CH Hoi. 2020. Deep learning for image super-resolution: A survey. IEEE transactions on pattern analysis and machine intelligence 43, 10 (2020), 3365--3387.

[26]

Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020. Learning texture transformer network for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5791--5800.

[27]

Hyunho Yeo, Chan Ju Chong, Youngmok Jung, Juncheol Ye, and Dongsu Han. 2020. NEMO: enabling neural-enhanced video streaming on commodity mobile devices. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.

Digital Library

[28]

Hyunho Yeo, Youngmok Jung, Jaehong Kim, Jinwoo Shin, and Dongsu Han. 2018. Neural adaptive content-aware internet video delivery. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 645--661.

[29]

YouTube. Accessed March 2, 2021. YouTube Live. https://www.youtube.com/live.

[30]

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European conference on computer vision (ECCV). 286--301.

Digital Library

[31]

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2472--2481.

[32]

Yinjie Zhang, Yuanxing Zhang, Yi Wu, Yu Tao, Kaigui Bian, Pan Zhou, Lingyang Song, and Hu Tuo. 2020. Improving quality of experience by adaptive video streaming with super-resolution. In IEEE INFOCOM 2020-IEEE Conference on Computer Communications. IEEE, 1957--1966.

Digital Library

Cited By

Chai BWu DChen JYang MWang ZHu M(2025)REM: Enabling Real-Time Neural-Enhanced Video Streaming on Mobile Devices Using Macroblock-Aware Lookup TableIEEE Transactions on Mobile Computing10.1109/TMC.2024.349644324:3(2085-2097)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3496443
Wang LSingh SChakareski JHajiesmaili MSitaraman R(2024)BONES: Near-Optimal Neural-Enhanced Video StreamingProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36560148:2(1-28)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3656014
Ghosh DVidyarthi AGautam CAgarwal RKant Agarwal KGupta PAlkhayyat AAlsulami M(2024)Elevating Personalized VOD Systems: Formal Analysis With VCR Actions and Trusted ComputingIEEE Transactions on Consumer Electronics10.1109/TCE.2024.339766270:4(6776-6786)Online publication date: Nov-2024
https://doi.org/10.1109/TCE.2024.3397662
Show More Cited By

Index Terms

Revisiting super-resolution for internet video streaming
1. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia streaming

Recommendations

Face image super-resolution through locality-induced support regression

In this paper we propose a novel face image super-resolution (SR) method named Locality-induced Support Regression (LiSR). Given a low-resolution (LR) input patch, we learn a mapping function between the local support LR and high-resolution (HR) patch ...
Deep Residual Attention Network for Spectral Image Super-Resolution
Computer Vision – ECCV 2018 Workshops
Abstract
Spectral imaging sensors often suffer from low spatial resolution, as there exists an essential tradeoff between the spectral and spatial resolutions that can be simultaneously achieved, especially when the temporal resolution needs to be ...
Block Adaptive Super Resolution Video Coding
PCM '09: Proceedings of the 10th Pacific Rim Conference on Multimedia: Advances in Multimedia Information Processing

Super resolution technique was first proposed for enhancing the image resolution, and then it was expanded to video sequence for obtaining a higher resolution video from low resolution input. Recently, super-resolution based video coding has emerged as ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

NOSSDAV '22: Proceedings of the 32nd Workshop on Network and Operating Systems Support for Digital Audio and Video

June 2022

92 pages

ISBN:9781450393836

DOI:10.1145/3534088

Program Chairs:
Zhisheng Yan
George Mason University
,
Michael Zink
University of Massachusetts Amherst
,
Yong Liu
New York University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Science and Technology Planning Project of Guangdong Province
National Natural Science Foundation of China under Grants

Conference

MMSys '22

Sponsor:

MMSys '22: 13th ACM Multimedia Systems Conference

June 17, 2022

Athlone, Ireland

Acceptance Rates

Overall Acceptance Rate 118 of 363 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
284
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)1

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chai BWu DChen JYang MWang ZHu M(2025)REM: Enabling Real-Time Neural-Enhanced Video Streaming on Mobile Devices Using Macroblock-Aware Lookup TableIEEE Transactions on Mobile Computing10.1109/TMC.2024.349644324:3(2085-2097)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3496443
Wang LSingh SChakareski JHajiesmaili MSitaraman R(2024)BONES: Near-Optimal Neural-Enhanced Video StreamingProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/36560148:2(1-28)Online publication date: 29-May-2024
https://dl.acm.org/doi/10.1145/3656014
Ghosh DVidyarthi AGautam CAgarwal RKant Agarwal KGupta PAlkhayyat AAlsulami M(2024)Elevating Personalized VOD Systems: Formal Analysis With VCR Actions and Trusted ComputingIEEE Transactions on Consumer Electronics10.1109/TCE.2024.339766270:4(6776-6786)Online publication date: Nov-2024
https://doi.org/10.1109/TCE.2024.3397662
Chai BChen JLuo ZWang ZHu MZhou YWu D(2024)SDSR: Optimizing Metaverse Video Streaming via Saliency-Driven Dynamic Super-ResolutionIEEE Journal on Selected Areas in Communications10.1109/JSAC.2023.334541842:4(978-989)Online publication date: Apr-2024
https://doi.org/10.1109/JSAC.2023.3345418
Chen QWen WQin J(2024)GlobalSRNeural Networks10.1016/j.neunet.2024.106686180:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106686
Li KGao GWang ZWu X(2023)Edge-Assisted Joint Rate Adaptation and Quality Enhancement for 360-Degree Video Streaming2023 IEEE 25th International Workshop on Multimedia Signal Processing (MMSP)10.1109/MMSP59012.2023.10337749(1-6)Online publication date: 27-Sep-2023
https://doi.org/10.1109/MMSP59012.2023.10337749
Liu LLi JXu HXue KXue J(2023)Efficient Real-time Video Conferencing with Adaptive Frame DeliveryComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2023.109918234:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.comnet.2023.109918
Gupta HJallepalli SMulchandani PTrasikar CManjesh CSwaminathan VPetrangeli S(2022)Towards Efficient Video Super Resolution for Faster Streaming2022 IEEE International Symposium on Multimedia (ISM)10.1109/ISM55400.2022.00031(153-154)Online publication date: Dec-2022
https://doi.org/10.1109/ISM55400.2022.00031

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten