Abstract
The cloud downloading scheme, first proposed by us in 2011, has effectively optimized hundreds of millions of users’ downloading experiences. Also, people start to build a variety of useful Internet services on top of cloud downloading. In brief, by using cloud facilities to download (and cache) the requested file from the “best-effort” Internet on behalf of the user, cloud downloading ensures the data availability and remarkably enhances the data delivery speed. Although this scheme seems simple and straightforward, designing a real-world cloud downloading system involves complicated and subtle trade-offs (between deployment cost and user experience) when serving a large number of users: 1) how to plan the cloud cache capacity to achieve a high and affordable cache hit ratio, 2) how to accelerate the data delivery from the cloud to numerous users, 3) how to handle the dense user requests for highly popular files, and 4) how to judge a potential downloading failure of the cloud. This paper addresses these design trade-offs from a practical perspective, based on big data from a nationwide commercial cloud downloading system, i.e., Tencent QQXuanfeng. Its running traces help us find reasonable design strategies and parameters, and its real performances confirm the efficacy of our design. Our study provides solid experiences and valuable heuristics for the developers of similar and relevant systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Liu Y, Guo L, Li F, Chen S. An empirical evaluation of battery power consumption for streaming data transmission to mobile devices. In Proc. the 19th ACM-MM, Nov.28–Dec.1, 2011, pp.473–482.
Chen G, Li Z. Peer-to-Peer Network: Structure, Application and Design. Tsinghua University Press, Sep. 2007. (in Chinese)
Liu Y, Xiao L, Liu X, Ni L M, Zhang X. Location awareness in unstructured peer-to-peer systems. IEEE Transactions on Parallel and Distributed Systems, 2005, 16(2): 163–174.
Wen Y, Zhu X, Rodrigues J, Chen C. Cloud mobile media: Reflections and outlook. IEEE Transactions on Multimedia, 2014, 16(4): 885–902.
Huang Y, Li Z, Liu G, Dai Y. Cloud download: Using cloud utilities to achieve high-quality content distribution for unpopular videos. In Proc. the 19th ACM-MM, Nov.28–Dec.1, 2011, pp.213–222.
Ao N, Xu Y, Chen C, Guo Y. Offline downloading: A non-traditional cloud-accelerated and peer-assisted file distribution service. In Proc. the 4th CyberC, Oct. 2012, pp.81–88.
Zhou Y, Fu Z, Chiu D M, Huang Y. An adaptive cloud downloading service. IEEE Transactions on Multimedia, 2013, 15(4): 802–810.
Li Z, Huang Y, Liu G, Wang F, Zhang Z L, Dai Y. Cloud Transcoder: Bridging the format and resolution gap between Internet videos and mobile devices. In Proc. the 22nd ACM NOSSDAV, Jun. 2012, pp.33–38.
Hu H, Wen Y, Chua T, Li X. Towards scalable systems for big data analytics: A technology tutorial. IEEE Access, 2014, 2: 652–687.
Podlipnig S, Böszörmenyi L. A survey of web cache replacement strategies. ACM Computing Surveys, 2003, 35(4): 374–398.
Zhang J, Izmailov R, Reininger D, Ott M. Web caching framework: Analytical models and beyond. In Proc. IEEE Workshop on Internet Applications (WIA), Jul. 1999, pp.132–141.
Arlitt M, Cherkasova L, Dilley J, Friedrich R, Jin T. Evaluating content management techniques for web proxy caches. ACM SIGMETRICS Performance Evaluation Review, 2000, 27(4): 3–11.
Wu D, Hou Y T, Zhu W, Zhang Y Q, Peha J M. Streaming video over the Internet: Approaches and directions. IEEE Transactions on Circuits and Systems for Video Technology, 2001, 11(3): 282–300.
Li Z, Cao J, Chen G. ContinuStreaming: Achieving high playback continuity of gossip-based peer-to-peer streaming. In Proc. the 22nd IEEE IPDPS, Apr. 2008.
Liu F, Shen S, Li B, Li B, Yin H, Li S. Novasky: Cinematic-quality VoD in a P2P storage cloud. In Proc. the 30th INFOCOM, Apr. 2011, pp.936–944.
Yin H, Liu X, Zhan T, Sekar V, Qiu F, Lin C, Zhang H, Li B. Design and deployment of a hybrid CDN-P2P system for live video streaming: Experiences with LiveSky. In Proc. the 17th ACM-MM, Oct. 2009, pp.25–34.
Wu C, Li B, Zhao S. On dynamic server provisioning in multichannel P2P live streaming. IEEE/ACM Transactions on Networking, 2011, 19(5): 1317–1330.
Li Z, Huang Y, Liu G, Wang F, Liu Y, Zhang Z L, Dai Y. Challenges, designs and performances of large-scale open-P2SP content distribution. IEEE Transactions on Parallel and Distributed Systems, 2013, 24(11): 2181–2191.
Li Z, Zhang T, Huang Y, Zhang Z L, Dai Y. Maximizing the bandwidth multiplier effect for hybrid Cloud-P2P content distribution. In Proc. the 20th ACM/IEEE IWQoS, Jun. 2012, pp.20:1–20:9.
Li Z, Wilson C, Xu T, Liu Y, Lu Z, Wang Y. Offline downloading in China: A comparative study. In Proc. the 15th ACM IMC, Oct. 2015.
Author information
Authors and Affiliations
Corresponding author
Additional information
Special Section on Networking and Distributed Computing for Big Data
This work is sponsored by the National Natural Science Foundation of China under Grant Nos. 61471217 and 61472266, the China Postdoctoral Science Foundation under Grant No. 2014M550735, and the CCF-Tencent Open Fund under Grant No. AGR20150201.
Rights and permissions
About this article
Cite this article
Li, ZH., Liu, G., Ji, ZY. et al. Towards Cost-Effective Cloud Downloading with Tencent Big Data. J. Comput. Sci. Technol. 30, 1163–1174 (2015). https://doi.org/10.1007/s11390-015-1591-5
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11390-015-1591-5