Patrick Pegus
Prashant Shenoy
ABSTRACT
In this paper, we present an open, flexible and realistic benchmarking platform named Video BenchLab to measure the performance of streaming media workloads. While Video BenchLab can be used with any existing media server, we provide a set of tools for researchers to experiment with their own platform and protocols. The components include a MediaDrop video server, a suite of tools to bulk insert videos and generate streaming media workloads, a dataset of freely available video and a client runtime to replay videos in the native video players of real Web browsers such as Firefox, Chrome and Internet Explorer. We define simple metrics that are able to capture the quality of video playback and identify issues that can happen during video replay. Finally, we provide a Dashboard to manage experiments, collect results and perform analytics to compare performance between experiments.
We present a series of experiments with Video BenchLab to illustrate how the video specific metrics can be used to measure the user perceived experience in real browsers when streaming videos. We also show Internet scale experiments by deploying clients in data centers distributed all over the globe. All the software, datasets, workloads and results used in this paper are made freely available on SourceForge for anyone to reuse and expand.
- ACM Multimedia Systems conference Dataset archive, http://traces.cs.umass.edu/index.php/Mmsys/MmsysGoogle Scholar
- Bienia, Christian, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. "The PARSEC benchmark suite: Characterization and architectural implications." In Proceedings of the 17th international ACM conference on Parallel architectures and compilation techniques, pp. 72--81. 2008. Google ScholarDigital Library
- Philip Bräunlich and Gerrit van Aaken -- HTML5 Video Player Comparison -- http://praenanz.de, last update 2014-07-09.Google Scholar
- Emmanuel Cecchet, Veena Udayabhanu, Timothy Wood and Prashant Shenoy -- BenchLab: An Open Testbed for Realistic Benchmarking of Web Applications -- Proceedings of 2nd USENIX Conference on Web Application Development (WebApps '11), June 15-16, 2011, Portland, OR. Google ScholarDigital Library
- Cherkasova, Ludmila, and Loren Staley. "Building a Performance Model of Streaming Media Applications in Utility Data Center Environment." In CCGRID, vol. 3, p. 52. 2003. Google ScholarDigital Library
- Chesire, Maureen, Alec Wolman, Geoffrey M. Voelker, and Henry M. Levy. "Measurement and Analysis of a Streaming Media Workload." In USITS, vol. 1, pp. 1--1. 2001. Google ScholarDigital Library
- The CloudSuite Media Streaming Benchmark, http://parsa.epfl.ch/cloudsuite/streaming.html, 2012.Google Scholar
- Ganglia Monitoring system - http://ganglia.sourceforge.net/.Google Scholar
- Google Video Quality Report, https://www.google.com/get/videoqualityreport/Google Scholar
- Phillipa Gill, Martin Arlitt, Zongpeng Li, and Anirban Mahanti-- YouTube Traffic Characterization: A View From the Edge -- IMC'07, October 24-26, 2007, San Diego, CA. Google ScholarDigital Library
- HTTP Archive specification (HAR) v1.2 - http://www.softwareishard.com/blog/har-12-spec/.Google Scholar
- M. Larson, M. Soleymani, M. Eskevich, P. Serdyukov, R. Ordelman, and G. Jones. "The community and the crowd: Developing large-scale data collections for multimedia benchmarking." IEEE Multimedia, (2012).Google Scholar
- C. Lee, P. Miodrag, and W. H. Mangione-Smith. "MediaBench: a tool for evaluating and synthesizing multimedia and communications systems." In Proceedings of the 30th annual ACM/IEEE international symposium on Microarchitecture, pp. 330--335. 1997. Google ScholarDigital Library
- C. Leung, and H. Ho-Shing Ip. "Benchmarking for content-based visual information search." In Advances in Visual Information Systems, pp. 442--456. Springer 2000. Google ScholarDigital Library
- Libav - Open source audio and video processing tools - http://libav.org/.Google Scholar
- S. Marchand-Maillet, and M. Worring. "Benchmarking image and video retrieval: an overview." In Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pp. 297--300, 2006. Google ScholarDigital Library
- MediaDrop - http://mediadrop.net/.Google Scholar
- Meeyoung Cha, Haewoon Kwak, Pablo Rodriguez, Yong-Yeol Ahn, and Sue Moon -- I Tube, You Tube, Everybody Tubes: Analyzing the World's Largest User Generated Content Video System -- IMC'07, October 24-26, 2007, San Diego, CA. Google ScholarDigital Library
- Li, Mingzhe, Mark Claypool, Robert Kinicki, and James Nichols. "Characteristics of streaming media stored on the Web." ACM Transactions on Internet Technology (TOIT) 5, no. 4: 601--626, 2005. Google ScholarDigital Library
- Gill, Phillipa, Martin Arlitt, Zongpeng Li, and Anirban Mahanti. "Youtube traffic characterization: a view from the edge." In Proceedings of the 7th ACM Internet Measurement Conference, pp. 15--28. 2007. Google ScholarDigital Library
- Rabbah, Rodric M., Ian Bratt, Krste Asanovic, and Anant Agarwal. "Versatility and versabench: A new metric and a benchmark suite for flexible architectures." MIT LCS Technical Report MIT-CSAIL-TR-2004-039, 2004.Google Scholar
- Jin, Shudong, and Azer Bestavros. "Gismo: a generator of internet streaming media objects and workloads." ACM SIGMETRICS Performance Evaluation Review 29, no. 3 pages 2--10 2001. Google ScholarDigital Library
- Jim Summers, Tim Brecht, Derek Eager, and Bernard Wong "Methodologies for generating HTTP streaming video workloads to evaluate web server performance", Proceedings of the 5th Annual International Systems and Storage Conference (SYSTOR '12). New York, NY, 2012. Google ScholarDigital Library
- Selenium - http://seleniumhq.org/.Google Scholar
- Slingerland, Nathan T., and Alan Jay Smith. "Design and characterization of the Berkeley multimedia workload." Multimedia Systems 8, no. 4: 315--327, 2002. Google ScholarDigital Library
- Standard Performance Evaluation Corporation (SPEC) Benchmarks, www.spec.orgGoogle Scholar
- Veloso, Eveline, Virgilio Almeida, Wagner Meira, Azer Bestavros, and Shudong Jin. "A hierarchical characterization of a live streaming media workload." In Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment, pp. 117--130. 2002. Google ScholarDigital Library
- Vimeo Creative Commons / Attribution Licensed video - http://vimeo.com/creativecommons/by.Google Scholar
- WebMetrics BrowserMob proxy - http://opensource.webmetrics.com/browsermob-proxy/.Google Scholar
- Michael Zink, Kyoungwon Suh, Yu Gu, and Jim Kurose, Characteristics of YouTube Network Traffic at a Campus Network - Measurements, Models, and Implications. Elsevier Computer Networks. Vol. 53, No. 4, March 2009. Google ScholarDigital Library
Index Terms
- Video BenchLab: an open platform for realistic benchmarking of streaming media workloads
Recommendations
Video BenchLab demo: an open platform for video realistic streaming benchmarking
MMSys '15: Proceedings of the 6th ACM Multimedia Systems ConferenceIn this demonstration, we present an open, flexible and realistic benchmarking platform named Video BenchLab to measure the performance of streaming media workloads. While Video BenchLab can be used with any existing media server, we provide a set of ...
BenchLab: an open testbed for realistic benchmarking of web applications
WebApps'11: Proceedings of the 2nd USENIX conference on Web application developmentWeb applications have evolved from serving static content to dynamically generating Web pages. Web 2.0 applications include JavaScript and AJAX technologies that manage increasingly complex interactions between the client and the Web server. Traditional ...
SPEC HPG benchmarks for high-performance systems
In this paper, we discuss the results and characteristics of the benchmark suites maintained by the Standard Performance Evaluation Corporation's (SPEC) High-Performance Group (HPG). Currently, SPECHPGhas two lines of benchmark suites for measuring ...
Comments