skip to main content
research-article

A High-Fidelity and Low-Interaction-Delay Screen Sharing System

Published: 20 May 2016 Publication History

Abstract

The pervasive computing environment and wide network bandwidth provide users more opportunities to share screen content among multiple devices. In this article, we introduce a remote display system to enable screen sharing among multiple devices with high fidelity and responsive interaction. In the developed system, the frame-level screen content is compressed and transmitted to the client side for screen sharing, and the instant control inputs are simultaneously transmitted to the server side for interaction. Even if the screen responds immediately to the control messages and updates at a high frame rate on the server side, it is difficult to update the screen content with low delay and high frame rate in the client side due to non-negligible time consumption on the whole screen frame compression, transmission, and display buffer updating. To address this critical problem, we propose a layered structure for screen coding and rendering to deliver diverse screen content to the client side with an adaptive frame rate. More specifically, the interaction content with small region screen update is compressed by a blockwise screen codec and rendered at a high frame rate to achieve smooth interaction, while the natural video screen content is compressed by standard video codec and rendered at a regular frame rate for a smooth video display. Experimental results with real applications demonstrate that the proposed system can successfully reduce transmission bandwidth cost and interaction delay during screen sharing. Especially for user interaction in small regions, the proposed system can achieve a higher frame rate than most previous counterparts.

References

[1]
Apple. 2016. Apple Airplay. Retrieved from http://www.apple.com/appletv/airplay/.
[2]
AnyDesk. 2016a. Homepage. Retrieved from http://anydesk.com/remote-desktop.
[3]
AnyDesk. 2016b. Remote Desktop Software Benchmark. Retrieved from http://anydesk.com/benchmark/anydesk-benchmark.pdf.
[4]
Ricardo A. Baratto, Leonard N. Kim, and Jason Nieh. 2005. THINC: A virtual display architecture for thin-client computing. ACM SIGOPS Operat. Syst. Rev. 39, 5 (2005), 277--290.
[5]
Surendar Chandra, Jacob T. Biehl, John Boreczky, Scott Carter, and Lawrence A. Rowe. 2012. Understanding screen contents for building a high performance, real time screen sharing system. In Proceedings of the 20th ACM International Conference on Multimedia. ACM, New York, NY, 389--398.
[6]
Yu-Chun Chang, Po-Han Tseng, Kuan-Ta Chen, and Chin-Laung Lei. 2011. Understanding the performance of thin-client gaming. In Proceedings of the 2011 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR). IEEE, 1--6.
[7]
Kuan-Ta Chen, Yu-Chun Chang, Hwai-Jung Hsu, De-Yu Chen, Chun-Ying Huang, and Cheng-Hsin Hsu. 2014. On the quality of service of cloud gaming systems. IEEE Trans. Multimed. 16, 2 (2014), 480--495.
[8]
Kuan-Ta Chen, Yu-Chun Chang, Po-Han Tseng, Chun-Ying Huang, and Chin-Laung Lei. 2011. Measuring the latency of cloud gaming systems. In Proceedings of the 19th ACM International Conference on Multimedia. ACM, New York, NY, 1269--1272.
[9]
Tihao Chiang and Ya-Qin Zhang. 1997. A new rate control scheme using quadratic rate distortion model. IEEE Trans. Circuits Syst. Video Technol. 7, 1 (1997), 246--250.
[10]
Sharon Choy, Bernard Wong, Gwendal Simon, and Catherine Rosenberg. 2012. The brewing storm in cloud gaming: A measurement study on cloud to end-user latency. In Proceedings of the 11th Annual Workshop on Network and Systems Support for Games. IEEE, 1--6.
[11]
Bernd Oliver Christiansen and Klaus Erik Schauser. 2002. Fast motion detection for thin client compression. In Proceedings of the 2002 Data Compression Conference. IEEE, 332--341.
[12]
Chromecast. 2016. Homepage. Retrieved from http://www.google.com/chrome/devices/chromecast/.
[13]
Lien Deboosere, Jeroen De Wachter, Pieter Simoens, Filip De Turck, Bart Dhoedt, and Piet Demeester. 2007. Thin client computing solutions in low-and high-motion scenarios. In Proceedings of the 3rd International Conference on Networking and Services, 2007. IEEE, 38--38.
[14]
Wenpeng Ding, Yan Lu, and Feng Wu. 2007. Enable efficient compound image compression in H. 264/AVC intra coding. In Proceedings of the IEEE International Conference on Image Processing (ICIP) 2007, Vol. 2. IEEE, 337--340.
[15]
DisplayCastSourcecode. 2016. Homepage. Retrieved from https://github.com/displayCast/.
[16]
FFmpeg. 2016. Homepage. Retrieved from http://ffmpeg.org/.
[17]
GamingAnywhere. 2016. GamingAnywhere: An Open Cloud Gaming System. Retrieved from https://github.com/chunying/gaminganywhere.
[18]
Patrick Haffner, Paul G. Howard, Patrice Simard, Yoshua Bengio, Yann Lecun, and others. 1998. High quality document image compression with DjVu. J. Electron. Imag. 7, 3 (1998), 410--425.
[19]
Mark Handley, Sally Floyd, Jitendra Padhye, and Jörg Widmer. 2002. TCP Friendly Rate Control (TFRC): Protocol Specification. Technical Report.
[20]
Chih-Fan Hsu, Tsung-Han Tsai, Chun-Ying Huang, Cheng-Hsin Hsu, and Kuan-Ta Chen. 2015. Screencast dissected: Performance measurements and design considerations. In Proceedings of the 6th ACM Multimedia Systems Conference. ACM, New York, NY, 177--188.
[21]
Chun-Ying Huang, Cheng-Hsin Hsu, Yu-Chun Chang, and Kuan-Ta Chen. 2013. GamingAnywhere: An open cloud gaming system. In Proceedings of the 4th ACM Multimedia Systems Conference. ACM, New York, NY, 36--47.
[22]
Michael Jarschel, Daniel Schlosser, Sven Scheuring, and Tobias Hofeld. 2011. An evaluation of QoE in cloud gaming based on subjective tests. In Proceedings of the 2011 5th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS). IEEE, 330--335.
[23]
LogMeIn. 2016. Homepage. Retrieved from https://secure.logmein.com/.
[24]
Yan Lu, Shipeng Li, and Huifeng Shen. 2011. Virtualized screen: A third element for cloud-mobile convergence. IEEE Trans. Multimed. 18, 2 (2011), 4--11.
[25]
Dan Miao, Jingjing Fu, Yan Lu, Shipeng Li, and Chang Wen Chen. 2013. Layered screen video coding leveraging hardware video codec. In Proceedings of the 2013 IEEE International Conference on Multimedia and Expo. IEEE, 1--6.
[26]
Dan Miao, Jingjing Fu, Yan Lu, Shipeng Li, and Chang Wen Chen. 2014. High frame rate screen video coding for screen sharing applications. In Proceedings of the 2014 IEEE International Symposium on Circuits and Systems. IEEE, 2157--2160.
[27]
Microsoft. 2016a. Remote Desktop Protocol (RDP). Retrieved from http://msdn.microsoft.com/en-us/library/aa383015(v=vs.85).aspx.
[28]
Microsoft. 2016b. Using Performance Monitor. Retrieved from https://technet.microsoft.com/en-us/library/cc749115.aspx.
[29]
NCast. 2016. Homepage. Retrieved from http://www.ncast.com/.
[30]
OnLive. 2016. Homepage. Retrieved from http://www.onlive.com/.
[31]
Zhaotai Pan, Huifeng Shen, Yan Lu, Shipeng Li, and Nenghai Yu. 2013. A low-complexity screen compression scheme for interactive screen sharing. IEEE Trans. Circuits Syst. Video Technol. 23, 6 (2013), 949--960.
[32]
Tristan Richardson, Quentin Stafford-Fraser, Kenneth R. Wood, and Andy Hopper. 1998. Virtual network computing. IEEE Trans. Internet Comput. 2, 1 (1998), 33--38.
[33]
Robert W. Scheifler and Jim Gettys. 1986. The X window system. ACM Trans. Graphics 5, 2 (1986), 79--109.
[34]
Huifeng Shen, Yan Lu, Feng Wu, and Shipeng Li. 2009. A high-performanance remote computing platform. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications, 2009. IEEE, 1--6.
[35]
Pieter Simoens, Paul Praet, Bert Vankeirsbilck, Jeroen De Wachter, Lien Deboosere, Filip De Turck, Bart Dhoedt, and Piet Demeester. 2008. Design and implementation of a hybrid remote display protocol to optimize multimedia experience on thin client devices. In Proceedings of the Australian Telecommunication Networks and Applications Conference, 2008 (ATNAC’08). IEEE, 391--396.
[36]
Software Systems Laboratory. 2016. THINC. Retrieved from http://systems.cs.columbia.edu/projects/thinc/.
[37]
StreamMyGame. 2016. Homepage. Retrieved from http://streammygame.com/smg/index.php.
[38]
StreamMyGame. 2012. Homepage. Retrieved from http://streammygame.com/smg/index.php.
[39]
Kheng-Joo Tan, Jia-Wei Gong, Bing-Tsung Wu, Dou-Cheng Chang, Hsin-Yi Li, Yi-Mao Hsiao, Yung-Chung Chen, Shi-Wu Lo, Yuan-Sun Chu, and Jiun-In Guo. 2010. A remote thin client system for real time multimedia streaming over VNC. In Proceedings of the 2010 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 992--997.
[40]
VideoLAN. 2016. x264. Retrieved from http://www.videolan.org/developers/x264.html.
[41]
TeamViewer. 2016. Homepage. Retrieved from http://www.teamviewer.com/.
[42]
TightVNC. 2016. Homepage. Retrieved from http://www.tightvnc.com/.
[43]
Shiqi Wang, Jingjing Fu, Yan Lu, Shipeng Li, and Wen Gao. 2012. Content-aware layered compound video compression. In Proceedings of the 2012 IEEE International Symposium on Circuits and Systems. IEEE, 145--148.
[44]
Wireshark. 2016. Homepage. Retrieved from https://www.wireshark.org/.
[45]
Wi-Fi Alliance. 2016. Wi-Fi CERTIFIED Miracast. Retrieved from http://www.wi-fi.org/discover-wi-fi/wi-fi-certified-miracast.
[46]
Jiyan Wu, Bo Cheng, Chau Yuen, Yanlei Shang, and Junliang Chen. 2015a. Distortion-aware concurrent multipath transfer for mobile video streaming in heterogeneous wireless networks. IEEE Trans. Mobile Comput. 14, 4 (2015), 688--701.
[47]
Jiyan Wu, Chau Yuen, Ngai-Man Cheung, Junliang Chen, and Chang Wen Chen. 2015b. Enabling adaptive high-frame-rate video streaming in mobile cloud gaming applications. IEEE Trans. Circuits Syst. Video Technol. 25, 12 (2015), 1988--2001.
[48]
Tao Zhang, Xun Guo, Yan Lu, Shipeng Li, Siwei Ma, and Debin Zhao. 2013. Arbitrary-sized motion detection in screen video coding. In Image Processing (ICIP), 2013 20th IEEE International Conference on. IEEE, 1943--1947.

Cited By

View all
  • (2024)Assessing Wireless Screen-Casting Services: A Comprehensive Quality of Experience Model2024 International Conference on Ubiquitous Communication (Ucom)10.1109/Ucom62433.2024.10695843(276-280)Online publication date: 5-Jul-2024
  • (2022)Investigating the pedagogies of screen‐sharing in contemporary learning environments—A mixed methods analysisJournal of Computer Assisted Learning10.1111/jcal.1264738:3(770-783)Online publication date: 8-Feb-2022
  • (2021)FLUID-XPProceedings of the 27th Annual International Conference on Mobile Computing and Networking10.1145/3447993.3483245(762-774)Online publication date: 25-Oct-2021
  • Show More Cited By

Index Terms

  1. A High-Fidelity and Low-Interaction-Delay Screen Sharing System

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Multimedia Computing, Communications, and Applications
    ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 12, Issue 3
    June 2016
    227 pages
    ISSN:1551-6857
    EISSN:1551-6865
    DOI:10.1145/2901366
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 May 2016
    Accepted: 01 February 2016
    Revised: 01 February 2016
    Received: 01 August 2015
    Published in TOMM Volume 12, Issue 3

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Interaction delay
    2. bandwidth consumption
    3. layered structure
    4. screen coding
    5. screen rendering
    6. video quality

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • Microsoft Research Asia

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)2
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 28 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Assessing Wireless Screen-Casting Services: A Comprehensive Quality of Experience Model2024 International Conference on Ubiquitous Communication (Ucom)10.1109/Ucom62433.2024.10695843(276-280)Online publication date: 5-Jul-2024
    • (2022)Investigating the pedagogies of screen‐sharing in contemporary learning environments—A mixed methods analysisJournal of Computer Assisted Learning10.1111/jcal.1264738:3(770-783)Online publication date: 8-Feb-2022
    • (2021)FLUID-XPProceedings of the 27th Annual International Conference on Mobile Computing and Networking10.1145/3447993.3483245(762-774)Online publication date: 25-Oct-2021
    • (2021)Full-reference Screen Content Image Quality Assessment by Fusing Multilevel Structure SimilarityACM Transactions on Multimedia Computing, Communications, and Applications10.1145/344739317:3(1-21)Online publication date: 22-Jul-2021
    • (2020)The identity authentication of Wi-Fi system based on network securityAnnals of Telecommunications10.1007/s12243-020-00793-0Online publication date: 28-Aug-2020
    • (2019)Low-Complexity Scalable Extension of the High-Efficiency Video Coding (SHVC) Encoding SystemACM Transactions on Multimedia Computing, Communications, and Applications10.1145/331318515:2(1-23)Online publication date: 5-Jun-2019
    • (2019)Learning Content-Specific Codebooks for Blind Quality Assessment of Screen Content ImagesSignal Processing10.1016/j.sigpro.2019.03.013Online publication date: Mar-2019
    • (2018)An In-depth Performance Analysis and Optimization for Android ScreencastProceedings of the 2nd International Conference on Computer Science and Application Engineering10.1145/3207677.3278016(1-6)Online publication date: 22-Oct-2018
    • (2018)Toward a Blind Quality Predictor for Screen Content ImagesIEEE Transactions on Systems, Man, and Cybernetics: Systems10.1109/TSMC.2017.267618048:9(1521-1530)Online publication date: Sep-2018
    • (2018)DTRMFuture Generation Computer Systems10.1016/j.future.2018.01.02683:C(293-302)Online publication date: 1-Jun-2018

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media