research-article

Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning

Authors:

Dongsu HanAuthors Info & Claims

SIGCOMM '20: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication

Pages 107 - 125

https://doi.org/10.1145/3387514.3405856

Published: 30 July 2020 Publication History

Abstract

Live video accounts for a significant volume of today's Internet video. Despite a large number of efforts to enhance user quality of experience (QoE) both at the ingest and distribution side of live video, the fundamental limitations are that streamer's upstream bandwidth and computational capacity limit the quality of experience of thousands of viewers.

To overcome this limitation, we design LiveNAS, a new live video ingest framework that enhances the origin stream's quality by leveraging computation at ingest servers. Our ingest server applies neural super-resolution on the original stream, while imposing minimal overhead on ingest clients. LiveNAS employs online learning to maximize the quality gain and dynamically adjusts the resource use to the real-time quality improvement. LiveNAS delivers high-quality live streams up to 4K resolution, outperforming WebRTC by 1.96 dB on average in Peak-Signal-to-Noise-Ratio on real video streams and network traces, which leads to 12%-69% QoE improvement for live stream viewers.

Supplementary Material

MP4 File (3387514.3405856.mp4)

20-minute talk about Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning presented at SIGCOMM' 20

Download
538.34 MB

References

[1]

Ant Media Server Official Website. https://antmedia.io/.

[2]

Apple HTTP Live Streaming Official Website. https://developer.apple.com/streaming/.

[3]

Apple Low-Latency HLS Specification. https://developer.apple.com/documentation/http_live_streaming/protocol_extension_for_low-latency_hls_preliminary_specification.

[4]

Chunked-encoded and Chunked-transferred CMAF Specification. https://www.akamai.com/us/en/multimedia/documents/white-paper/low-latency-streaming-cmaf-whitepaper.pdf.

[5]

Cisco Visual Networking Index Report. http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/complete-white-paper-c11-481360.pdf.

[6]

DASH Industry Forum Official Website. https://dashif.org/.

[7]

Encoding Video at the Edge with Intel® Xeon® Processors. https://builders.intel.com/docs/networkbuilders/encoding-video-at-the-edge-with-intel-xeon-processors.pdf.

[8]

FCC Broadband Bandwidth Measurement. https://www.fcc.gov/reports-research/reports/measuring-broadband-america/raw-data-measuring-broadband-america-eighth.

[9]

libWebRTC Official Github Repository. https://github.com/aisouard/libwebrtc.

[10]

Live Streaming Statistics about Twitch and Facebook. https://www.theverge.com/2020/1/9/21058907/twitch-youtube-mixer-facebook-live-streaming-numbers-growth-q4.

[11]

Real-Time Messaging Protocol (RTMP) Specification. https://wwwimages2.adobe.com/content/dam/acom/en/devnet/rtmp/pdf/rtmp_specification_license_1.0.pdf.

[12]

Real Time Streaming Protocol (RTSP) Specification. https://tools.ietf.org/html/rfc2326.

[13]

Real-time Transport Protocol (RTP) Specification. https://tools.ietf.org/html/rfc3550.

[14]

Streamlabs Q2 2019 Live Streaming Industry Report. https://blog.streamlabs.com/q2-2019-7e8039277b11.

[15]

Twitch Broadcasting Guidelines. https://stream.twitch.tv/encoding/.

[16]

Twitch Most Popular Game Category on February 2020. https://twitchtracker.com/games/rating. Last accessed Feb. 6, 2020.

[17]

Twitch Official Website. https://www.twitch.tv/.

[18]

Twitch Statistics about Streamer Channel (Saddummy). https://www.twitch.tv/saddummy.

[19]

Twitch Statistics Report by Influencer MarketingHub. https://influencermarketinghub.com/twitch-statistics/.

[20]

Twitch Stream Dataset (Escape from Tarkov). https://www.twitch.tv/stylishnoob4/.

[21]

Twitch Stream Dataset (Fornite). https://www.twitch.tv/nickmercs/videos.

[22]

Twitch Stream Dataset (Just Chatting). https://www.twitch.tv/pokimane/videos.

[23]

Twitch Stream Dataset (League of Legends). https://www.twitch.tv/riotgames/videos.

[24]

Twitch Stream Dataset (World of Warcraft). https://www.twitch.tv/method/.

[25]

WebM Official Website. https://www.webmproject.org/.

[26]

WebRTC Official Website. https://webrtc.org/.

[27]

Wowza 2019 Live-Streaming Advancements Report. https://www.wowza.com/blog/streaming-advancements-2019.

[28]

Wowza Live Streaming Guidebook. https://www.wowza.com/uploads/images/The-Complete-Guide-to-Live-Streaming-Wowza.pdf.

[29]

Wowza WebRTC Streaming Guidebook. https://www.wowza.com/low-latency/webrtc-streaming-software.

[30]

Youtube Live Encoding Guidelines. https://support.google.com/youtube/answer/2853702?hl=en.

[31]

Youtube Live Streaming Official Website. https://www.youtube.com/live.

[32]

Youtube Video Dataset (Food). https://www.youtube.com/watch?v=oLbRhd7p0Ts.

[33]

Youtube Video Dataset (Live Event). https://www.youtube.com/watch?v=aVZngNyEsjg.

[34]

Youtube Video Dataset (Pod Cast). https://www.youtube.com/watch?v=CpQZRLxJE5M&=&index=14.

[35]

Youtube Video Dataset (Sport). https://www.youtube.com/watch?v=ppb7hmQA6sU.

[36]

Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.

[37]

Ghufran Baig, Jian He, Mubashir Adnan Qureshi, Lili Qiu, Guohai Chen, Peng Chen, and Yinliang Hu. 2019. Jigsaw: Robust live 4k video streaming. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.

Digital Library

[38]

Gaetano Carlucci, Luca De Cicco, Stefan Holmer, and Saverio Mascolo. 2016. Analysis and Design of the Google Congestion Control for Web Real-Time Communication (WebRTC). In Proceedings of the 7th International Conference on Multimedia Systems (MMSys '16). Article Article 13, 12 pages.

Digital Library

[39]

M. Dasari, A. Bhattacharya, S. Vargas, P. Sahu, A. Balasubramanian, and S. R. Das. 2020. Streaming 360° Videos using Super-resolution. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM).

[40]

J. Deng, F. Cuadrado, G. Tyson, and S. Uhlig. 2015. Behind the game: Exploring the twitch streaming platform. In 2015 International Workshop on Network and Systems Support for Games (NetGames). 1--6.

[41]

Florin Dobrian, Vyas Sekar, Asad Awan, Ion Stoica, Dilip Joseph, Aditya Ganjam, Jibin Zhan, and Hui Zhang. 2011. Understanding the Impact of Video Quality on User Engagement. SIGCOMM Comput. Commun. Rev. 41, 4 (Aug. 2011), 362--373.

Digital Library

[42]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a deep convolutional network for image super-resolution. In European conference on computer vision. Springer, 184--199.

[43]

Zhengfang Duanmu, Kai Zeng, Kede Ma, Abdul Rehman, and Zhou Wang. 2016. A quality-of-experience index for streaming video. IEEE Journal of Selected Topics in Signal Processing 11, 1 (2016), 154--166.

[44]

Sadjad Fouladi, John Emmons, Emre Orbay, Catherine Wu, Riad S. Wahby, and Keith Winstein. 2018. Salsify: Low-Latency Network Video through Tighter Integration between a Video Codec and a Transport Protocol. In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). Renton, WA, 267--282.

Digital Library

[45]

Sadjad Fouladi, Riad S. Wahby, Brennan Shacklett, Karthikeyan Vasuki Balasubramaniam, William Zeng, Rahul Bhalerao, Anirudh Sivaraman, George Porter, and Keith Winstein. 2017. Encoding, Fast and Slow: Low-Latency Video Processing Using Thousands of Tiny Threads. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). Boston, MA, 363--376.

Digital Library

[46]

Jeremy Fowers, Kalin Ovtcharov, Michael Papamichael, Todd Massengill, Ming Liu, Daniel Lo, Shlomi Alkalay, Michael Haselman, Logan Adams, Mahdi Ghandi, et al. 2018. A configurable cloud-scale DNN processor for real-time AI. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA). IEEE, 1--14.

Digital Library

[47]

F. Fund, C. Wang, Y. Liu, T. Korakis, M. Zink, and S. S. Panwar. 2013. Performance of DASH and WebRTC Video Services for Mobile Users. In 2013 20th International Packet Video Workshop. 1--8.

[48]

Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading, Mass. http://www.worldcat.org/search?qt=worldcat_org_all&q=0201633612

Digital Library

[49]

Alain Hore and Djemel Ziou. 2010. Image quality metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition. IEEE, 2366--2369.

Digital Library

[50]

Pan Hu, Rakesh Misra, and Sachin Katti. 2019. Dejavu: Enhancing Videoconferencing with Prior Knowledge. In Proceedings of the 20th International Workshop on Mobile Computing Systems and Applications (HotMobile '19). 63--68.

Digital Library

[51]

Mehdi Kaytoue, Arlei Silva, Loïc Cerf, Wagner Meira, and Chedy Raïssi. 2012. Watch Me Playing, I am a Professional: A First Study on Video Game Live Streaming. In Proceedings of the 21st International Conference on World Wide Web (WWW '12 Companion). 1181--1188.

Digital Library

[52]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image superresolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1646--1654.

[53]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[54]

Royson Lee, Stylianos I Venieris, Lukasz Dudziak, Sourav Bhattacharya, and Nicholas D Lane. 2019. MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors. In The 25th Annual International Conference on Mobile Computing and Networking. 1--16.

Digital Library

[55]

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 136--144.

[56]

Siwei Ma, Wen Gao, and Yan Lu. 2005. Rate-distortion analysis for H. 264/AVC video coding and its application to rate control. IEEE transactions on circuits and systems for video technology 15, 12 (2005), 1533--1544.

Digital Library

[57]

Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. 2017. Neural Adaptive Video Streaming with Pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). Association for Computing Machinery, 197--210.

Digital Library

[58]

Ravi Netravali, Anirudh Sivaraman, Somak Das, Ameesh Goyal, Keith Winstein, James Mickens, and Hari Balakrishnan. 2015. Mahimahi: Accurate Record-and-Replay for HTTP. In Proceedings of the USENIX Annual Technical Conference (ATC). 417--429.

[59]

KarinePires andGwendalSimon. 2014. Dashintwitch:Adaptive bitrate streaming in live game streaming platforms. In Proceedings of the 2014 Workshop on Design, Quality and Deployment of Adaptive Video Streaming. 13--18.

[60]

Karine Pires and Gwendal Simon. 2015. You Tube Live and Twitch: A Tour of User-Generated Live Streaming Systems. In Proceedings of the 6th ACM Multimedia Systems Conference (MMSys '15). 225--230.

Digital Library

[61]

Devdeep Ray, Jack Kosaian, K. V. Rashmi, and Srinivasan Seshan. 2019. Vantage: Optimizing Video Upload for Time-Shifted Viewing of Social Live Streams. In Proceedings of the ACM Special Interest Group on Data Communication (SIGCOMM '19). 380--393.

Digital Library

[62]

Haakon Riiser, Paul Vigmostad, Carsten Griwodz, and P~l Halvorsen. 2013. Commute path bandwidth traces from 3G networks: analysis and applications. In Proceedings of the 4th ACM Multimedia Systems Conference. ACM, 114--118.

Digital Library

[63]

Shai Shalev-Shwartz et al. 2012. Online learning and online convex optimization. Foundations and Trends® in Machine Learning 4, 2 (2012), 107--194.

[64]

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1874--1883.

[65]

Assaf Shocher, Nadav Cohen, and Michal Irani. 2018. "zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3118--3126.

[66]

Iraj Sodagar. 2011. The mpeg-dash standard for multimedia streaming over the internet. IEEE multimedia 18, 4 (2011), 62--67.

Digital Library

[67]

Mikko Uitto. 2016. Energy consumption evaluation of H. 264 and HEVC video encoders in high-resolution live streaming. In 2016 IEEE 12th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob). IEEE, 1--7.

[68]

Yiding Wang, Weiyan Wang, Junxue Zhang, Junchen Jiang, and Kai Chen. 2019. Bridging the Edge-Cloud Barrier for Real-time Advanced Vision Analytics. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19). USENIX Association, Renton, WA. https://www.usenix.org/conference/hotcloud19/presentation/wang

[69]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.

Digital Library

[70]

David H Wolpert. 1996. The lack of a priori distinctions between learning algorithms. Neural computation 8, 7 (1996), 1341--1390.

[71]

Hyunho Yeo, Sunghyun Do, and Dongsu Han. 2017. How will Deep Learning Change Internet Video Delivery?. In Proceedings of the 16th ACM Workshop on Hot Topics in Networks. ACM, 57--64.

Digital Library

[72]

Hyunho Yeo, Youngmok Jung, Jaehong Kim, Jinwoo Shin, and Dongsu Han. 2018. Neural Adaptive Content-aware Internet Video Delivery. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). Carlsbad, CA, 645--661.

[73]

Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. 2015. A control-theoretic approach for dynamic adaptive video streaming over HTTP. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 325--338.

Digital Library

[74]

Cong Zhang and Jiangchuan Liu. 2015. On Crowdsourced Interactive Live Streaming: A Twitch.Tv-Based Measurement Study. In Proceedings of the 25th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV '15). 55--60.

Digital Library

[75]

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. 2018. Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2472--2481.

[76]

Zhengdong Zhang and Vivienne Sze. 2017. FAST: A framework to accelerate super-resolution processing on compressed videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 19--28.

Cited By

Dritsas ETrigka M(2025)Exploring the Intersection of Machine Learning and Big Data: A SurveyMachine Learning and Knowledge Extraction10.3390/make70100137:1(13)Online publication date: 7-Feb-2025
https://doi.org/10.3390/make7010013
Chai BWu DChen JYang MWang ZHu M(2025)REM: Enabling Real-Time Neural-Enhanced Video Streaming on Mobile Devices Using Macroblock-Aware Lookup TableIEEE Transactions on Mobile Computing10.1109/TMC.2024.349644324:3(2085-2097)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3496443
Wu HWu DGong P(2025)SR-ABR: Super Resolution Integrated ABR Algorithm for Cloud-Based Video StreamingIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.34464499:1(87-98)Online publication date: Feb-2025
https://doi.org/10.1109/TETCI.2024.3446449
Show More Cited By

Index Terms

Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning
1. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia streaming
2. Networks
  1. Network algorithms
    1. Control path algorithms
      1. Network resources allocation

Recommendations

Higher quality live streaming under lower uplink bandwidth: an approach of super-resolution based video coding
NOSSDAV '21: Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video

With the growing popularity of live streaming, high video quality and low latency with limited uplink bandwidth have become a significant challenge. In this study, we propose Live Super-Resolution Based Video Coding (LiveSRVC), a novel video uploading ...
HTTP/2-Based Methods to Improve the Live Experience of Adaptive Streaming
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

HTTP Adaptive Streaming (HAS) is today the number one video technology for over-the-top video distribution. In HAS, video content is temporally divided into multiple segments and encoded at different quality levels. A client selects and retrieves per ...
Live Streaming as Co-Performance: Dynamics between Center and Periphery in Theatrical Engagement

Live streaming is a highly participatory form of performance, involving various types of audience participation such as liking, commenting, and gifting. But how do streamers and audiences collaborate to deliver live streaming performances? We approach ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGCOMM '20: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication

July 2020

814 pages

ISBN:9781450379557

DOI:10.1145/3387514

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SIGCOMM '20

Sponsor:

SIGCOMM

SIGCOMM '20: Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication

August 10 - 14, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 462 of 3,389 submissions, 14%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

109
Total Citations
View Citations
4,853
Total Downloads

Downloads (Last 12 months)474
Downloads (Last 6 weeks)31

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dritsas ETrigka M(2025)Exploring the Intersection of Machine Learning and Big Data: A SurveyMachine Learning and Knowledge Extraction10.3390/make70100137:1(13)Online publication date: 7-Feb-2025
https://doi.org/10.3390/make7010013
Chai BWu DChen JYang MWang ZHu M(2025)REM: Enabling Real-Time Neural-Enhanced Video Streaming on Mobile Devices Using Macroblock-Aware Lookup TableIEEE Transactions on Mobile Computing10.1109/TMC.2024.349644324:3(2085-2097)Online publication date: Mar-2025
https://doi.org/10.1109/TMC.2024.3496443
Wu HWu DGong P(2025)SR-ABR: Super Resolution Integrated ABR Algorithm for Cloud-Based Video StreamingIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.34464499:1(87-98)Online publication date: Feb-2025
https://doi.org/10.1109/TETCI.2024.3446449
Drucker RBalasubramanian AGandhi A(2025)Investigating WebRTC BBR as an alternative to GCC for live video streaming2025 17th International Conference on COMmunication Systems and NETworks (COMSNETS)10.1109/COMSNETS63942.2025.10885565(225-232)Online publication date: 6-Jan-2025
https://doi.org/10.1109/COMSNETS63942.2025.10885565
Sivaraman VKarimi PVenkatapathy VKhani MFouladi SAlizadeh MDurand FSze VVanbever LZhang I(2024)GeminoProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691857(569-590)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691857
Chen BYan ZZhang YYang ZNahrstedt KVanbever LZhang I(2024)LiFteRProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691855(533-548)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691855
Cheng YZhang ZLi HArapin AZhang YZhang QLiu YDu KZhang XYan FMazumdar AFeamster NJiang JVanbever LZhang I(2024)GRACEProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691854(509-531)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691854
He ZYang YLi SQiu LDai DYang Y(2024)VIGOR: Reviving Cloud Gaming SessionsProceedings of the ACM on Networking10.1145/36964032:CoNEXT4(1-20)Online publication date: 25-Nov-2024
https://dl.acm.org/doi/10.1145/3696403
Siam SAhn HLiu LAlam SShen HCao ZShroff NKrishnamachari BSrivastava MZhang M(2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1145/3690639
Yu EZhou JLi ZTyson GLi WZhang XXu ZXie G(2024)Mustang: Improving QoE for Real-Time Video in Cellular Networks by Masking JitterACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367239920:9(1-23)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3672399
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten