Amy Csizmar Dalal
Abstract
While subjective measurements are the most natural for assessing the user-perceived quality of a media stream, there are issues with their scalability and their context accuracy. We explore techniques to select application-layer measurements, collected by an instrumented media player, that most accurately predict the subjective quality rating that a user would assign to a stream. We consider three feature subset selection techniques that reduce the number of features (measurements) under consideration to ones most relevant to user-perceived stream quality. Two of the three techniques mathematically consider stream characteristics when selecting measurements, while the third is based on observation. We apply the reduced feature sets to two nearest-neighbor algorithms for predicting user-perceived stream quality. Our results demonstrate that there are clear strategies for estimating the quality rating that work well in specific circumstances such as video-on-demand services. The results also demonstrate that neither of the mathematically-based feature subset selection techniques identify a single set of features that is unambiguously influential on user-perceived stream quality, but that ultimately a combination of retransmitted and/or lost application-layer packets is most accurate for predicting stream quality.
- Ashmawi, W., Guerin, R., Wolf, S., and Pinson, M. H. 2001. On the impact of policing and rate guarantees in Diff-Serv networks: A video streaming application perspective. In Proceedings of the ACM SIGCOMM Data Communications Festival. Google Scholar
Digital Library
- Calyam, P., Sridharan, M., Mandrawa, W., and Schopis, P. 2004. Performance measurement and analysis of H.323 traffic. In Proceedings of the Passive and Active Measurement Conference.Google Scholar
- Carson, M. and Santay, D. 2003. NIST Net: a Linux-based network emulation tool. SIGCOMM Comput. Comm. Rev. 33, 3, 111--126. Google ScholarDigital Library
- Clark, A. D. 2001. Modeling the effects of burst packet loss and recency on subjective voice quality. In Proceedings of the IP Telephony Workshop.Google Scholar
- Comscore, Inc. 2008. YouTube attracts 100 million U.S. online video viewers in October 2008. http://www.comscore.com/press/release.asp?press=2616.Google Scholar
- Csizmar Dalal, A. 2009. Revisiting a QoE assessment architecture six years later: Lessons learned and remaining challenges. In Proceedings of the 3rd International Workshop on Advanced Architectures and Algorithms for Internet Delivery and Applications.Google ScholarCross Ref
- Csizmar Dalal, A., Bouchard, A., Cantor, S., Guo, Y., and Johnson, A. 2010. Predicting video stream quality in real time using objective stream state measurements. Tech. rep., Carleton College.Google Scholar
- Csizmar Dalal, A., Musicant, D. R., Olson, J., McMenamy, B., Benzaid, S., Kazez, B., and Bolan, E. 2007. Predicting user-perceived quality ratings from streaming media data. In Proceedings of the IEEE International Conference on Communications.Google Scholar
- Csizmar Dalal, A. and Olson, J. 2007. Feature selection for prediction of user-perceived streaming media quality. In Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems.Google Scholar
- Csizmar Dalal, A. and Perry, E. 2003. A new architecture for measuring and assessing streaming media quality. In Proceedings of the Passive and Active Measurement Conference.Google Scholar
- Csizmar Dalal, A. and Purrington, K. 2005. Discerning user-perceived media stream quality through application-layer measurements. In Proceedings of the 1st International Conference on Multimedia Services Access Networks.Google Scholar
- Dunham, M. H. 2002. Data Mining: Introductory and Advanced Topics. Prentice Hall. Google ScholarDigital Library
- French, H., Lin, J., Phan, T., and Csizmar Dalal, A. 2011. Real time video QoE analysis of RTMP streams. In Proceedings of the 30th IEEE International Performance Computing and Communications Conference (IPCCC). Google ScholarDigital Library
- Fu, A., Keough, E., Lau, L. Y. H., Ratanamahatana, C. A., and Wong, R. 2008. Scaling and time warping in time series querying. VLDB J. 17, 4, 899--921. Google ScholarDigital Library
- ITU. 1998. The Emodel, a computational model for use in transmission planning. Recommendation G.107.Google Scholar
- Hand, D., Mannila, H., and Smyth, P. 2001. Principles of Data Mining. MIT Press, Cambridge, MA. Google ScholarDigital Library
- Keogh, E. and Ratanamahatana, C. 2005. Exact indexing of dynamic time warping. Knowl. Inform. Syst. 7, 3, 358--386. Google ScholarDigital Library
- Krosnick, J. A. and Fabrigar, L. R. 1997. Survey Measurement and Process Quality. Wiley-Interscience, 141--165.Google Scholar
- Li, M., Claypool, M., and Kinicki, R. 2002. MediaPlayer versus RealPlayer: A comparison of network turbulence. In Proceedings of the ACM SIGCOMM Internet Measurement Workshop. Google ScholarDigital Library
- Li, M., Li, F., Claypool, M., and Kinicki, R. 2005. Weather forecasting - predicting performance for streaming video over wireless LAN s. In Proceedings of the ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video. Google ScholarDigital Library
- Mohamed, S. and Rubino, G. 2002. A study of real-time packet video quality using random neural networks. IEEE Trans. Circ. Syst. Video Techn. 12, 12, 1071--1083. Google ScholarDigital Library
- Nichols, J., Claypool, M., Kinicki, R., and Li, M. 2004. Measurement of the congestion responsiveness of Windows streaming media. In Proceedings of the ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video. Google ScholarDigital Library
- Ratanamahatana, C. A. and Keogh, E. 2004. Everything you know about dynamic time warping is wrong. In Proceedings of the 3rd Workshop on Mining Temporal and Sequential Data, in Conjunction with the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.Google Scholar
- Sakoe, H. and Chiba, S. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans Acoust. Speech Signal Process. 26, 43--49.Google ScholarCross Ref
- Serral-Gracia, R., Yannuzzi, M., Marin-Tordera, E., Masip-Bruin, X., and Sanchez, S. 2010. Quality of experience enforcement in wireless networks. In Proceedings of the 8th International Conference on Wired/Wireless Internet Communications. E. Osipov, A. Kassler, T. Bohnert, and X. Masip-Bruin, Eds., Lecture Notes in Computer Science, vol. 6074, 180--191. Google ScholarDigital Library
- Tan, P., Steinbach, M., and Kumar, V. 2005. Introduction to Data Mining. Addison-Wesley. Google ScholarDigital Library
- Tang, R., William, M., Shaw, J., and Vevea, J. L. 1999. Towards the identification of the optimal number of relevance categories. J. Amer. Soc. Inform. Sci. 50, 3, 254--264. Google ScholarDigital Library
- Vlachos, M., Hadjieleftheriou, M., Gunopulos, D., and Keogh, E. 2003. Indexing multidimensional time-series with support for multiple distance measures. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM Press, New York, NY, 216--225. Google ScholarDigital Library
- Wang, Y and Claypool, M. 2005. RealTracer: Tools for measuring the performance of RealVideo on the internet. Kluwer Multimedia Tools Appl. 27, 3. Google ScholarDigital Library
- Wang, Y, Claypool, M., and Zuo, Z. 2001. An empirical study of RealVideo performance across the Internet. In Proceedings of the International Memory Workshop.Google Scholar
- Wolf, S. and Pinson, M. H. 1999. Spatial-temporal distortion metrics for in-service quality monitoring of any digital video system. In Proceedings of the SPIE International Symposium on Voice, Video, and Data Communications.Google Scholar
- Yoon, H., Yang, K., and Shahabi, C. 2005. Feature subset selection and feature ranking for multivariate time series. IEEE Trans. Knowl. Data Engin. 17, 9. Google ScholarDigital Library
Recommendations
Comparative Evaluation of User Perceived Quality Assessment of Design Strategies for HTTP-based Adaptive Streaming
HTTP-based Adaptive Streaming (HAS) is the dominant Internet video streaming application. One specific HAS approach, Dynamic Adaptive Streaming over HTTP (DASH), is of particular interest, as it is a widely deployed, standardized implementation. Prior ...
IPTV quality assessment system
LANC '12: Proceedings of the 7th Latin American Networking ConferenceDue to the Increasing deployment of real-time multimedia services like IPTV and videoconferencing, the Internet has new Challenges. These new real-time Applications require a reliable performance of the network so as to provide a good Quality of Service ...
Modelling of spatio-temporal interaction for video quality assessment
Video services have appeared in the recent years due to advances in video coding and convergence to IP networks. As these emerging services mature, the ability to deliver adequate quality to end-users becomes increasingly important. However, the ...
Comments